Developer's Guide

Developer Environment

The project was developed in Eclipse Indigo EE with Hibernate Tools and Subversive (SVN) plugins. However Hibernate Tools was used only to generate object-relational mappings(ORM) and are no longer necessary. In below sections we described how to check-out project from Totem SVN, attach missing libraries, build WAR file of DBEL Web Interface and JAR with console utility.

Project Checkout

Project is available in two versions as a web interface and console utility. If you look into https://svn.cern.ch/reps/totem/trunk/dbel you will find DBELWS (DBEL Web Server) and DBELC (DBEL Console). DBELWS folder contains project named DBEL which provides web interface and core functionality which is required also by console utility. The folder DBELC contains console utility project called DBELC. If you want to develop web interface (DBEL project) you can checkout only DBELWS folder, but be aware that console utility (DBELC project) has references to DBEL project. If you want to develop console utility you should checkout both projects.

The tutorial below is prepared for Eclipse developers if you prefer development in other environments look at Ant build scripts section.

Project checkout in Eclipse

In order to checkout DBEL project using Eclipse follow described steps.

1. Click File button in menu. Then go thorugh Import -> SVN -> Project from SVN
2. In the Checkout from SVN form fill URL: https://svn.cern.ch/reps/totem/trunk/dbel/DBELWS and type your credentials
3. Then press Next and Finish
4. In form Check Out As press Finish and fill your Eclipse Secure Storage password
5. The project should appear in your workspace

Buildpath Configuration

The project should have two Java build path problems, the first error occurred because the configuration file with Totem Database account details is secret and has not been synchronized with SVN repository.

In order to solve the error do the following steps:

1. Create folder credentials in the project root folder outside the Eclipse.

~/workspace/DBEL$ mkdir credentials

1. Refresh the project. Click right mouse button on the name of the project in Project Explorer and press Refresh.

The main build problem should be solved, but there are also problem with missing JSP library, look at the first screen

The easiest solution is just adding Tomcat v7.0 Runtime Environment to Eclipse:

1. Press Window button in menu bar
2. Go through Preferences -> Server -> Runtime Environment tabs.
3. Click Add button.
4. Choose Apache Tomcat v7.0 and click Next
5. Fill in your own Tomcat installation directory and press Finish

The problem with missing JSP library should be solved then automatically.

Ant build files

There are three Ant build scripts for DBEL and DBELC projects. They can be found in SVN at:

1. DBELWS/build.xml
2. DBELWS/main_build.xml
3. DBELC/build.xml

DBELWS/build.xml is basic build file which allows to compile (compile ant target) DBEL project and generate WAR file (war ant target). The generated WAR file does not contain DBEL.zip file with console utility and it's configuration. The main purpose of creating this file is to support DBEL project compilation and deployment without DBELC project.

After project checkout update build.xml with path to your Tomcat's/lib directory as the value of property TOMCAT_LIB similar to example:

<property name="TOMCAT_LIB" value="/usr/share/tomcat7/lib"/>

If you have not changed project structure after checkout simply type in console:

~/workspace/DBEL$ ant compile

in order to compile DBEL project.

If you want to create WAR file type:

~/workspace/DBEL$ ant war

DBELC/build.xml allows to compile DBELC project and generate JAR file with version number and build date in MANIFEST.MF. If you want to change version number update version.num property

<property name="version.num" value="1.00"/>

Take into account that DBELC project has references on DBEL project and requires it to successful compilation. You do not have to change the build file after project checkout because all required JAR files are supplied with the project.

DBELWS/main_build.xml is dedicated for developers who want to build both projects DBEL and DBELC. The script provides below build targets:

1. compile_server - compile only DBEL project
2. compile_console - compile only DBELC project
3. war - generate WAR file without DBEL.zip file with console utility
4. jar - generate JAR file for console utility
5. all - prepares deployment version of DBEL project

All target compiles both DBEL and DBELC project, prepares DBEL.jar with version and date of build in MANIFEST.MF with console utility, packs DBEL.jar with console utility configuration in cli.properties in DBEL.zip file (which can be downloaded by users using web interface) and creates WAR file with DBEL project and DBEL.zip file which is ready for deployment on application server.

You must use -f option to build projects using main_build.xml because it is not default ant build file name, look at the example below.

If you have any problem with compilation check whether your project structure responds the example:

~/workspace$ ls . DBEL DBELC
.:
DBEL  DBELC
DBEL:
bindings  build.xml  credentials  main_build.xml  src  WebContent
DBELC:
bin  build.xml  cli.properties  lib  src

Now we will prepare deployment version of DBEL and DBELC projects:

~/workspace$ cd DBEL
~/workspace/DBEL$ ant -f main_build.xml all
Buildfile: /home/mateusz/workspace/DBEL/main_build.xml
clean:
compile_server:
    [mkdir] Created dir: /home/mateusz/workspace/DBEL/build/classes
    [javac] Compiling 102 source files to /home/mateusz/DBEL/build/classes
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
compile_console:
    [javac] Compiling 3 source files to /home/mateusz/workspace/DBEL/build
all:
    [mkdir] Created dir: /home/mateusz/workspace/DBEL/out
    [mkdir] Created dir: /home/mateusz/workspace/DBEL/temp
     ...
   [delete] Deleting directory /home/mateusz/workspace/DBEL/temp/ch
     [copy] Copying 242 files to /home/mateusz/workspace/DBEL/temp
     [copy] Copying 105 files to /home/mateusz/workspace/DBEL/temp/WEB-INF
     [copy] Copying 37 files to /home/mateusz/workspace/DBEL/temp/WEB-INF/classes
      [war] Building war: /home/mateusz/workspace/DBEL/out/DBEL.war
   [delete] Deleting directory /home/mateusz/workspace/DBEL/temp
BUILD SUCCESSFUL
Total time: 16 seconds

We strongly recommend to use described ant scripts instead of ones generated by Eclipse. The presented ant scripts were handwritten and tuned for project needs, created JAR have only required classes and referenced packages, so you are getting for example 5MB JAR with console utility instead of 28MB one created by Eclipse. Moreover Eclipse copies unnecessary configuration files with database credentials (hibernate.properties) which must be deleted manually from JAR content before deployment. Using described Ant build files solves such problems automatically

Eclipse WAR Generation

1. Before generating WAR-file you should edit DBEL's configuration files adding credentials to Totem Database and CERN Authentication Web Service.
2. Create file hibernate.properties in folder credentials. The file should have following content:

   hibernate.connection.username=TYPE_TOTEM_DATABASE_ACCOUNT_LOGIN
   hibernate.connection.password=TYPE_TOTEM_DATABASE_ACCOUNT_PASSWORD
   hibernate.connection.url=jdbc:oracle:thin:@TYPE_HOST_SERVER:TYPE_PORT_NUMBER:TYPE_SID
   

   For more detailed information about hibernate.properties file please refer to Administrator's Guide
3. The last configuration step will be CERN Authentication Web Service account configuration. Fill serviceUsername and servicePassword in /WebContent/WEB-INF/security-app-context.xml

<beans:bean id="myAuthenticationProvider" class="ch.cern.totem.dbel.security.CERNAuthenticationProvider">
      <beans:property name="serviceUsername" value="TYPE_SERVICE_ACCOUNT_LOGIN"/>
      <beans:property name="servicePassword" value="TYPE_SERVICE_ACCOUT_PASSWORD"/>
      <beans:property name="serviceGroup" value="zj"/>
      <beans:property name="namespaceLocalization" value="https://winservices-soap.web.cern.ch/winservices-soap/Generic/Authentication.asmx"/>
      <beans:property name="serviceEndpoint" value="https://winservices-soap.web.cern.ch/winservices-soap/Generic/Authentication.asmx"/>
   </beans:bean>

For more detailed information about CERN Authentication please refer to Administrator's Guide.

1. To create new WAR file click right mouse button on the project name, press Export -> WAR file.
2. Now you can easily deploy DBEL via Tomcat App Manager.

Eclipse JAR Generation

1. In order to develop console utility - DBELC you should also attach DBEL project to the workspace. Import DBELC project to the workspace from the SVN repository. The project can be found at https://svn.cern.ch/reps/totem/trunk/dbel/DBELC. You should not encounter any "Eclipse problems".
2. Click with right mouse button on the DBELC project, press Export -> Java -> Runnable JAR (If you do not see any run configuration, Cancel exporting form and simply run RemoteClient.java as a Java Application. Now repeat export steps, fill in Runnable JAR File Export form (we strongly suggest to export jar with DBEL.jar name) and press FINISH. If you see any warnings press OK.
3. Delete hibernate.properties file if it was exported to the jar. Create file cli.properties with the following content:

dbel.host=TYPE_SERVER_HOST\:TYPE_PORT_NUMBER

and put correct server host and port number.

1. Pack the DBEL.jar file and cli.properties together in DBEL.zip file.
2. If you want to publish your latest modifications replace Tomcat's /webapps/DBEL/DBEL.zip with packed jar. The name of zip file has to remain the same.
3. Above step assumes that you have deployed WAR with DBEL before.

Developing project in different environments

We were developing DBEL in Eclipse IDE, because it allows us to develop and test the project very effectively. However, you can use your preferable developer environment. We are not able to provide individual introductory tutorial how to build DBEL project in every possible IDE, so decided to write a self explanatory tutorial how to build under console.

We assume that you have installed Java 1.6 required by the project and console utilities: wget, ant, svn.

1. Create directory where you want to develop the application

   mkdir Workspace
   cd Workspace/
   

2. Download Tomcat and unpack application server to get JSP libraries required to future compilation.

   wget http://mirror.speednetwork.de/apache/tomcat/tomcat-7/v7.0.20/bin/apache-tomcat-7.0.20.zip
   unzip apache-tomcat-7.0.20.zip 
   rm apache-tomcat-7.0.20.zip 
   mv apache-tomcat-7.0.20 tomcat7

3. Checkout DBEL Web Server project to DBEL directiory. We advice to use this direcory in order to omit possible build problems.

   svn co svn+ssh://[CERN_username]@svn.cern.ch/reps/totem/trunk/dbel/DBELWS DBEL

4. Checkout DBEL Console project to DBELC directory.

   svn co svn+ssh://[CERN_username]@svn.cern.ch/reps/totem/trunk/dbel/DBELC DBELC
   cd DBEL

5. Update build files in DBEL directory changing TOMCAT_LIB property to absolute path of tomcat7/lib directory in your Workspace.

   pico main_build.xml 
   pico build.xml 

6. Create credentials directory, because it is required by build scripts.

   mkdir credentials

7. Point 2 and 3 from chapter Eclipse WAR Generation (above)
8. Compile both projects, export jar file, create zip with console utility and it's configuration, generate WAR file with the web interface simply typing:

   ant -f main_build.xml all

9. If the build was successful the WAR file can be found in Workspace/DBEL/out directory.

If you encountered any build errors check whether you exactly followed presented steps. Keep in mind that above instruction does not cover any configuration steps like providing credentials to Totem Database or application server configuration. If you are going to deploy the application read second and third step from Eclipse WAR Generation section in this document, for more detailed information about application server configuration refer to Administrator's Guide.

Logging level

DBEL project uses log4j logger as a default logger for Hibernate and Spring. Logger configuration for DBEL and DBELC can be found at DBEL/src/log4j.xml and DBELC/src/log4j.xml respectively. Server side application uses INFO logging level and logs to files. We provided sample configuration from our final project version below:

<appender name="rolling-file" class="org.apache.log4j.RollingFileAppender">
        <param name="file" value="~/DBEL.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="4"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%d [%t] %-5p %l - %m%n"/>
        </layout>
</appender>
<root>
        <priority value ="info" />
        <appender-ref ref="console" />
        <appender-ref ref="rolling-file" />
</root>

Console version uses less restrictive logging policy. Logging level was lowered to WARN in order not to write Spring initialization logs on user's console and logging to files was turned off. If end-user wants to see a stacktrace in case of exceptions he can use -vv (--veryVerbose) to quickly diagnose the problem.

Feel free to adapt above configuration to satisfy your requirements. Refer to log4j documentation if you need more detailed information about logger and it's configuration.

Technology Overview

One of the core assumptions according to DBEL architecture says "Do not use too sophisticated technologies". We took this phase into account seriously and choose technologies basing on their usability and learning curve. Technologies and application reasons are listed below.

1. Hibernate 3.6 The framework provides access to Totem Database and transparent transitions between SQL and Java objects. It was decided to use non-sql approach in querying Totem Database and benefit from static control of types owing to Hibernate Criteria. Object-Relational mappings were generated via Hibernate Tools Eclipse Plugin.
2. JAXB It takes care of Java object serialization to XML files. The appropriate Java objects were decorated with JAXB annotations to tune serialization process and lower the size of XML file. JAXB's stream approach to serialization is memory friendly, efficient and completely hides the fact that XML is used.
3. Spring Remoting It can be regarded as easier and more scalable Java RMI equivalent. DBEL uses this framework to exchange objects between console utility and application server.
4. Apache Commons CLI The library is used by console utility to parse command line arguments and print help.
5. Spring Security It introduces security to DBEL infrastructure. If user wants to get access to any DBEL resource (query web interface) or download data via console utility he will be authenticated with CERN nice login. All security layer in DBEL requires one XML file(Spring Security configuration) and one Java class. The framework is completely separated from other technologies and if you do not want to modify the security model you can leave it unchanged.
6. SAAJ Above library is used to create, exchange and read SOAP messages. DBEL connects to CERN Authentication Web Service, which is SOAP based, to authenticate users.
7. JSP The technology has the smallest learning curve from available Java web pages frameworks. It does not require any configuration and is very effective in simple applications(up to several web-pages).
8. JQuery It is a Javascript library. We used it to create unfold list with measurement names and data validation forms in query web interface. The solution results from JSP application.
9. Class Mexer The library is a Java agent to enable memory usage measurements. DBEL has internal memory protection mechanism. If memory for results of Totem Database query exceeds 256MB the query is dropped out. Memory protection mechanism is regarded as a good practice at CERN and LHC Logging Database developers advise us to introduce it to DBEL.

Under the hood

The main goal of DBEL was to provide users an easy access to the data stored in TOTEM database.
There was also a need to separate users from physical, raw access to DB.
The causes of this was:

• access and retreivng data from DB are complex tasks
• there are many programing languages for client application and many ways of fetching data (SQL, HQL, ORMs)
• users must be aware of the structure of DB
• users konowledge is insufficient, it may damage the database
• by mistake some data can be updated or even more - deleted
• users actions can slow down the database
• users have to transform the retrieved data to the desirable form

Taking into account all these facts, the one reader solution was adopted.
Advantages of such architecture are:

• completely prevent the user from direct access to the database
• unified data output format (defined by XML scheme)
• simple use in clients application
• because there is only one reader, the DB can be optimized for this reader
• optimized queries
• ability to use modern technology
• provide greater data security

Complementary solution to the only reader is the only wirter (for TOTEM database this tool is provided by DBPop).

How does it work?
Simply user via console application or web interface specifies the query's parameters. Then the query is processed by application, which is deployed on Tomcat server. This app queries the database, extracts data and creates output file, which goes to the end user. But in details how does it work? Which technologies are used?
Let's start with the user web interface. User specifies his query via webpsite created in JavaServletPages (JSP). Form validation on this page, calendars for timestamps input and tree with measurements was created using jQuery.
The rest of the site is written using simple HTML&CSS. When user click Submit the following actions are performed (JavaScript):

• script checks whether at least one Measurement was selected
• another scripts validate all fields
• the DynaTree (tree with measurement) changes HTML list (<ul>...<li>) to the inputs (type = hidden)
• the control is passed to the QueryServlet

QueryServlet has two main methods: doGet and doPost. They are called when the query form was passed respectively using GET or POST methods. In DBEL the form is passed via POST method. Next in QueryServlet the following actions are performed:

• QueryParameterInterpreter (method interpret) extracts all specified parameters from servlet request, validates extracted data and fills fields of the QueryParameterInterpreter with relevant data
• GeneralMeasurementManager is created; this object is responsible for getting data from database and transform retrieved data to GeneralContainerJaxb
• using method getData from GeneralMeasurementManager a GeneralContainerJaxb is returned
• response is prepared
• results of the query (GeneralContainerJaxb) is written into OutputStream of the response
• user get na result.xml file to download (window in browser "Open/Save as")

Described above method interpret from QueryParameterInterpreter do the following actions:

• search all possible parameters of DBEL in parameter map (from form), which stores pair: name of parameter - value of parameter
• checks whether the value of paramter is correct (eg. is a number, is greater then minimal value...)
• if parameter was specified and the value is correct, the proper field of the object is filled with given value

Method getData accepts as parameter QueryParameterInterpreter object. Then data from this object is unpacked and overloaded version of getData is called.
In this function the connection to database is opened. DBEL uses Hibernate to accessing to and operating on DB (Oracle). So in getData method Hibernate's session is opened (HibernateUtil class is resposible for this). Then if description option is specified, the description is retrieved from database (the way of getting data is the same as in retreiving measurement data (the only difference is ommision of predicting query size, which is described below). The description is written to previously created GeneralContainerJaxb object.
Next the transaction on DB is being opened, the predictQuerySize is called. This method trys to retreieve probationary data (the same specification, but number of results restricted to the max equals AVERAGE_RESULT_PROBE_COUNT). Then using classmexer library the usage of memory is counted per one single result (data about one measurement). DBEL checks also number of results for specified query. If this number multiplied by usage of memory per one result exceeds MAX_QUERY_SIZE, TooMemoryConsumingQuery exception is thrown.
If size of the query is permitted the DBEL parameters (eg. measurements names, timestamps etc.) are transformed into Criteria (Hibernate tools for specifying the query) - method prepareCriteria is resposible for this action. Then using Hibernate's criteria.list() data is retrieved from database new GeneralContainerJaxb is created, which is initialized with this data. At the end the GeneralContainerJaxb is appended to the GeneralJaxbContainer with description (or empty if no description), and the second one is returned.
GeneralContainerJaxb is a class which can be serialized into XML files using JAXB library (there are appropriate JAXB annotations in this class). Such container sores list of rows (each row contains some part of the result of the query). These some parts are added to the rows using method translate from TranslatorJaxb. Method translate transforms object retrieved from database (mapped by Hibernate to the appropriate mapping class) to corresponding objects, which can be serialized using JAXB.
Method save from GeneralContainerJaxb marshal list of the rows to the XML form, which is written to the output stream.
But what about the console?
Console version of DBEL to be able to perform similar action as web version must obtain GeneralMeasurementManager. It is implemented using SpringRemoting. This mechanism makes object distributed. But let's back to begin. RemoteClient (main class of console version) begin with loading properties such: host address (server with DBEL), username, password from file cli.properties. Then ApplicationContext is created. Next arguments of application are parsed, and checked whether they are correct (Apache Commons CLI). If so new security context is created (used for authentication) and GeneralMeasurementManager is obtained from DBEL server (app on Tomcat). In this way we can operate on the object which physically is on server, this solves following tasks:

• credentials to database don't have to be stored on local disk
• console version doesn't contain Hibernate mappings and JAXB annotated classes and many others classes
• main action is performed on the server side not on the client side
• the way of using DBEL under the hood (console and web versions is more less the same)

The rest of console version works similar as the web version, but user can specifiy also the output file (on local dsystem).

Package structure

• DBEL: common & web version
  • ch.cern.totem.dbel - GeneralMeasurementManager described in the section...
  • ch.cern.totem.dbel.common - interface IGeneralMeasurementManager for SpringRemoting; exceptions; dictionary classes with names of columns and specified parameters; QueryParameterInterpreter described in the section Under The Hood; TimestampConverter
  • ch.cern.totem.dbel.database.datatypes - classes used in Hibernate mappings, which describes user defined types from Oracle Database
  • ch.cern.totem.dbel.database.tables - tables mappings used by Hibernate
  • ch.cern.totem.dbel.database.views - views from database used by Hibernate
  • ch.cern.totem.dbel.hibernate - class for managing Hibernate sessions
  • ch.cern.totem.dbel.jaxb - classes describing simple types in JAXB, exeception connected with JAXB, TranslatorJaxb and GeneralContainerJaxb described in the section Under The Hood; QueryDescriptionJaxb used for descriptions of the query
  • ch.cern.totem.dbel.jaxb.tables - classes corresponding to the Hibernate mappings (which can be serialized using JAXB)
  • ch.cern.totem.dbel.measurements - all stuff needed to configure measurement names (load from XML)
  • ch.cern.totem.dbel.security - class for CERN authentication
  • ch.cern.totem.dbel.servlet - servlets used by DBEL
  • WebContent - XML configs and all JSP pages
• DBELC: console version
  • ch.cern.totem.dbel.console - argument parser and interpreter (CLI), class for storing parameters, main class - RemoteClient

Future Development

At this time project DBEL has been completed. It provides interface to query Totem Database and save results in XML files via console utility and web interface. However, the project is not closed and in our opinion there are two suggested ways of DBEL future development:

• providing object access to measurements
• enabling other Totem Database tables to query

Object access to measurements is the easiest and the most efficient to end users because they do not need to parse XML on their own and deal with files. It can be done as follows:

1. Download data using DBEL console utility
2. Open the file with data or read DBEL's STDOUT with *SAX parser*(we strongly recommend this kind of parser, because output files can be too big for DOM based approach e.g. 130MB for 1000 measurements.)
3. Deserialize objects from XML and provide them to user.

The presented solution looks simple and you should not encounter technology problems during implementation. The main drawback of this solution is the fact that you should provide one library for a supported programming language. Alternative solution may include introduction of distributed programming technology like ICE or Corba, but after taking all things into consideration it turns out to be very complex and unusable for DBEL project. The other solutions like application of SWIG http://www.swig.org/ or JNI after deeper research reveals that they describe how to use C++ existing code in Java, but not in the opposite direction!

DBEL was developed from scratch with Totem Database at development level according to CERN naming regulations. The database stored data only in tables connected with T18GeneralMeasurements. It was decided that DBEL will be responsible for extracting this data. In the future other tables may store data and it will turn out that DBEL should be extended to support new range of tables. We suggest to do the following steps in such case:

1. Basing on existing Hibernate mappings objects (they cover all tables) create their JAXB representations.
2. Design general interface to support your queries and implement it.
3. Create web interface as a parallel way of making your queries.
4. Expose object responsible for making queries to console utility via Spring Remoting.
5. Upgrade console utility to support your modifications.

Take presented problems and their solutions only as a suggestion. Of course you may have other requirements which we could not have taken into account writing this section.

-- JakubSzymanek - 06-Sep-2011