How to test the online alignment procedure on a single node

Due to the necessity of having DIM, the online alignment tests can only run in the online environment, e.g. on a plus or HLT node.

Setup environment

Since subversion is only available on plus nodes, perform these steps there. They are only needed once.

   $> SetupProject --build-env AlignmentOnline v10r2
   $> getpack -p anonymous AlignmentOnline/PyGeneticOnline head
   $> getpack -p anonymous PRConfig head

Use a new shell to build the packages

   $> cd $User_release_area/AlignmentOnline_v10r2
   $> make -j 5

Make sure to svn update and recompile before any tests, as improvements are still being made.

Find a suitable machine to run on

Since plus nodes are quite limited in terms of CPU capacity, HLT nodes are generally quite well suited, provided they are available and idle. To find a node, the following utility can be used:

   $> /group/online/dataflow/scripts/farmStatus &

Using the mouse or keyboard, navigate to subfarm that contains green nodes and press [Return] to see the status of the nodes. Look at the CPU column and pick an idle node. If no nodes are idle, press [Return] to exit the subfarm window.

Please note that machines with hostnames hltX0Y, with X = {a-f} and Y = {0-6} are the farm controllers and are not intended to run tasks. The actual farm nodes have hostnames that look like hltX0YZZ, with X = {a-f}, Y = {0-6} and Z = {01-20}.

Make sure the data is available

The online alignment test will start a number of worker processes that will read data from file(s). These file(s) must be available on the node that is used, so either copy the files to the node or put them in a folder on NFS. A small sample of test data files is located on hlte0902 at /localdisk/hlt1/test_align.

The worker processes can either each read a different file, or all read the same file. If they all read the same file, the same events will be "sent" to the iterator multiple times, which might not be desirable.

When separate files are required, make sure they are all from the same run and a number of files at least as large as the number or worker processes is present in the data directory.

A script to split a raw file into a larger number of chuncks that all contain N different events is attached to the page.

Setup the running environment and run the test

Once you have found a node, log in to it, setup the running environment and run.

   $> SetupProject AlignmentOnline
   $> export PYTHONPATH=/home/raaij/pydim/lib/python2.7/site-packages:/scratch/jenkins/benchmark/python:$PYTHONPATH
   $> cd $User_release_area/AlignmentOnline_v10r2/PRConfig/scripts
   $> python AlignOnlineTest.py --numa --nodes=2 --workers=10 --task-log=log/align.log --viewers PythonGenetic

For an explanation of the arguments to AlignOnlineTest.py, run

   $> python AlignOnlineTest.py --help

The final argument should be the name of a python module in PRConfig/python/AlignmentTests/TestModules that contains the required information to start the iterator and analyzer processes.

Looking at the (intermediate) results

A new temporary directory is created for each test and printed in the log that goes to the test log file and stdout. This directory should contain all intermediate files, results and conditions.

Cleaning up

If the test completes successfully, all processes will be terminated, except the LogViewer, which can be exited using Ctrl+C. If the test is interupted, not all processes might be terminated, clean them up using:

   $> killall Gaudi.exe

The temporary directory is not removed and should be manually removed once the files it contains are no longer needed for debugging or analysis.

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2015-03-24 - AndriiUsachov
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback