TrackerDQMOfflineTools < Sandbox

---+ Tracker !DQM offline tools

%TOC%

---++ Introduction

The plots produced by the DQM modules are visualized using a dedicated !DQM GUI, that permits to navigate easily the numerous plots needed to properly monitor the detector status and the quality of data reconstruction. %BR%
A quick introduction to the DQM GUI functionalities and usage can be found in [[CMSPublic.WorkBookDQM][this twiki]], or in the [[%PUBURL%/CMS/TrackerOfflineShiftTutorialSessions/Tracker_DQM_Tutorial.pdf][Tracker DQM Tutorial slides]]. In summary, the !DQM GUI, is organized in different workspaces, each relative to a different DPG (i.e !SiStrip, Pixel) and POG (i.e Tracking), that can be inspected (and also modified) in a user-friendly web interface. %BR%
The hierarchy structure of the different workspaces reflect the structure of the relative DQM files, plus some "special" folders meant to give a quick summary view of the full content created at the GUI level to ease data monitoring.%BR%
In particular each workspace contains a "QuikCollection" folder with the fundamental plots needed to asses the status of that particular subsystem (i.e !SiStrip, Pixel...) and a "Layout" folder with a more extended expert-view. Also "special" workspaces are created to summarize the status of the full !CMS detector, as an example the "Summary" workspace contains basic plots showing quality test results on the different !CMS subsytems, or the "Shift" workspaces where central shifters can check a basic collection of plots from each subsystem.%BR%

The !DQM !GUI is the main tool for data monitoring and certification, used extensively by the Tracker Offline shift crew and central !DQM shifters. The Tracker DQM team is in charge of maintaining the content of !SiStrip, Pixel and Tracking workspaces as well as the "Shift" and "Summary" worksapaces for the Online and Offline flavour of the GUI, while the Central DQM team is in charge of the software implementation of the GUI. %BR%
In particular, for what concerns the GUI development, the Tracker DQM is responsible to decide the content of the "QuickCollection" and "Layout" folder for the three workspaces above, plus the "Shift" workspace for the Offline and Online DQM GUI, and to provide a proper description of each plot.

---++ Some useful links
   * The github link to the !DQM !GUI code: [[https://github.com/dmwm/deployment]].
   * You need to have your GRID certificate installed in your browser ( [[CMS.DQMGUIGridCertificate][instructions]]).
   * [[https://cmsweb.cern.ch/dqm/offline][offline]] and [[https://cmsweb.cern.ch/dqm/online][online]] !DQM Graphical User Interface (!GUI) (GRID certificate needed, [[CMS.DQMGUIGridCertificate][instructions to get the certificate]])
   * For more detailed and complete documentation to develop !DQM applications please refer to the central !DQM page [[CMS.DQMGuiForUsers][here]]%BR%
   * If interested also in Online !DQM development, please refer to this [[CMS.DQMOnlineTesting][page]].%BR%
   * Documentation to run DQM packages can be found here [[CMS.DQMOffline][DQMOffline]] a basic guide for beginners can be found here: [[CMS.BasicInstructionsforTrackerDQMDevelopers][BasicInstructionsforTrackerDQMDevelopers]] %BR%
   * If you need any support to develop Tracker specific !DQM modules, please email hn-cms-trk-dqm@cern.ch.%BR%
   * If you find any problem on running standard !DQM configurations, please email the central !DQM hn-cms-dqmDevel@cern.ch. %BR%

---++ Goal of this page:

This twiki page collects a set of basic instructions to perform the standard operation of development/maintainace of the DQM GUI relevant for the Tracker DQM team. In particular the content of the "Quick Collection" and "Layout" folder for the !SiStrip, Pixel and Tracking workspaces are decided and modified, if needed, by the Tracker DQM team, the same applies to the "Shift" workspace. Moreover descriptions of plots shown in the twiki, have to be provided by Tracker DQM too. %BR%
The modifications to the GUI usually performed by the Tracker DQM are moving (or removing) plots in the "special" workspaces and folders, adding decriptions and customization to the plots. This page contains instructions to perform and test these standard developments.

#GUIandLayouts

---++ 1. The GUI and the layouts

Very detailed informations can be found in [[CMS.DQMGuiForUsers][Sandbox.DQMGuiForUsers]] and won't therefore be rewritten here. We strongly recommend newcomers to go through the whole twiki.
On this twiki, one can find information about what is the !DQM !GUI, what are the different flavors of !GUI, how the release cycle works, and how to set up your own !DQM !GUI.

#OnlineGUITest
---++ 2. Test of the online DQM GUI

In order to be able to reproduce the way the data is fed in the online flavor or the !DQM !GUI, a dedicated machine has been set up for the tracker group: fu-c2f11-21-01.
All the information related to this machine (how to access it, how to feed a run in the GUI, etc) is described thoroughly in the [[CMS.DQMOnlineTesting][DQMOnlineTesting]] twiki. More tracker related information can be found [[#PlayOwnGui][here]].

#HowToList
---++ 3. How to...

This section will give straightforward instructions to be able to manipulate the GUI, improving the layouts and is basically a "ready to use" summary of what can be found in the twiki referred above. If it is the first time you play with those tools, or if you would like to learn more about what you are doing, we strongly recommend to read the first twiki referenced above : [[CMS.DQMGuiForUsers][Sandbox.DQMGuiForUsers]]%BR%%BR%
The following two steps are mandatory to set up your own !DQM !GUI and to be able to perform and test new changes:
---+++ Create your own new GUI

%TWISTY{mode="div" showlink="Create your own new GUI" hidelink=" Hide 'Create your own new GUI'" remember="off" showimgleft="%ICONURLPATH{toggleopen-small}%" hideimgleft="%ICONURLPATH{toggleclose-small}%" class="twikiHelp"}%
Go on lxplus and download the code in a temporary directory:

<verbatim>
     ssh <username>@lxplus.cern.ch  #login to lxplus
     /bin/bash
     export PATH=$PATH:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin
     mkdir -p /tmp/$USER/testGui
     cd /tmp/$USER/testGui     #we will work in tmp to avoid issue with non-closed GUI (more secure)
     git clone git://github.com/dmwm/deployment.git
     $PWD/deployment/Deploy -A slc6_amd64_gcc493 -r "comp=comp" -R comp@<LATESTTAG> -t MYDEV -s "prep sw post" $PWD dqmgui/bare  #for the latest tag, have a look here: https://github.com/dmwm/deployment/releases
     source current/apps/dqmgui/128/etc/profile.d/env.sh
     </verbatim>

Choose your favorite flavor:

<verbatim>
     $PWD/current/config/dqmgui/manage -f dev start "I did read documentation"
     $PWD/current/config/dqmgui/manage -f online start "I did read documentation"
     $PWD/current/config/dqmgui/manage -f offline start "I did read documentation"
     $PWD/current/config/dqmgui/manage -f relval start "I did read documentation"
     </verbatim>

Now you can browse to your private DQM GUI in your favorite browser, by filling in your host name in the following address: (If you don't know the host name, type the command ==hostname== to find out.)

<verbatim>
     hostname:8060/dqm/dev
     hostname:8070/dqm/online-dev
     hostname:8080/dqm/offline
     hostname:8081/dqm/relval
     </verbatim>

Important note: if you are not at P5, you will need to tunnel to lxplus (see section [[#TunnelingLxplus][Tunneling to lxplus]]).%BR%
%ICON{help}% Since you will be working in a /tmp/ directory, this means all you do is linked to a specific UI. Therefore, if you plan to make a long work don't forget to write down the UI you're working on (e.g. lxplus054)... and also to make quick backups since /tmp/ folder is meant to be cleaned frequently.
 %ENDTWISTY%

---+++ Feed the GUI with a DQM file

%TWISTY{mode="div" showlink="Feed the GUI with a DQM file" hidelink=" Hide 'Feed the GUI with a DQM file'" remember="off" showimgleft="%ICONURLPATH{toggleopen-small}%" hideimgleft="%ICONURLPATH{toggleclose-small}%" class="twikiHelp"}%
---++++!! Step1: Find data %BR%

   According to the flavor you want to run on, find the !ROOT file you are interested in (!StreamExpress, !ZeroBias, ...)

<verbatim>
      Offline: https://cmsweb.cern.ch/dqm/offline/data/browse/ROOT/OfflineData
      Relval: https://cmsweb.cern.ch/dqm/relval/data/browse/ROOT/
      Online: https://cmsweb.cern.ch/dqm/online/data/browse/Original 
      </verbatim>
---++++!! Step 2: Download the file to lxplus

   To download directly the file on lxplus, you can write:

<verbatim>
      grid-proxy-init -rfc -bits 1024
      myproxy=`grid-proxy-info -path`
      wget --certificate=$myproxy --private-key=$myproxy --ca-certificate=$myproxy https://cmsweb.cern.ch/dqm/and_the_path_you_are_interested_in
      </verbatim>

      %ICON{help}% Due to certain [[CMS.DQMGuiForUsers#Loading_some_data][convention]], the file name should be version 1 (DQM_V000<b>1</b>), please change the version (name) of your file before uploading it to the GUI (see Step 3).%BR%
---++++!! Step 3: Upload your DQM file to your GUI (according to the flavor you want to run on)

<verbatim>
     source current/apps/dqmgui/128/etc/profile.d/env.sh

     visDQMUpload http://<your_hostname>.cern.ch:8060/dqm/dev FILENAME.root
     visDQMUpload http://<your_hostname>.cern.ch:8070/dqm/online-dev FILENAME.root
     visDQMUpload http://<your_hostname>.cern.ch:8080/dqm/offline FILENAME.root
     visDQMUpload http://<your_hostname>.cern.ch:8081/dqm/relval FILENAME.root
     </verbatim>

After some time it should be uploaded to your GUI, you can select the uploaded run like in the production GUI (for example, in the dev and online flavors of the GUI, you will need to click on the ==Live== button to be able to chose your run). Same applies if you want to switch between different files you uploaded.%BR%
Note: you can upload several files for a single runs (e.g. Pixel and !SiStrip). It will still appear as a single run in the GUI but will contain both files.

%ENDTWISTY%

Now that your own !DQM !GUI is woking, you can go ahead editing it. The next point will present an overview of the !DQM !GUI from the tracker point of view. Then, a list of "How to's" follows with specific instructions for tasks you could be asked to do.
---+++ Edit the GUI - the structure

%TWISTY{mode="div" showlink="Edit the layouts" hidelink=" Hide 'Edit the layouts'" remember="off" showimgleft="%ICONURLPATH{toggleopen-small}%" hideimgleft="%ICONURLPATH{toggleclose-small}%" class="twikiHelp"}%
Once your custom !GUI installed, go in the dqmgui folder: ==cd deployment/dqmgui==.%BR%
There, you will find several files :%BR%
 - ==server-conf-*.py==, where you can find which layouts are loaded in which GUI flavor. This is particularly important if you wish to create a new GUI file, or to understand if the layouts you just changed will be seen in a specific flavor of the GUI.%BR%
 - ==workspaces-*.py==, this file shows the Layouts for the Quick Collection, i.e. what will be displayed when the workspace is opened for the first time.%BR%
 - ==layouts==, the folder containing all the configuration files for each workspace layout.%BR%
 - ==style=, the folder containing all the customisation for the plots in the !DQM !GUI, such as logarithmic axes, grid option, drawing lines... %BR%

%ICON{help}% The dqmgui folder is a *development folder*. It means that the change you will make there will *not* be directly applied to your test server. To apply those changes, you will have to re-deploy, using the same commands as before:

<verbatim>
     $PWD/deployment/Deploy -A slc6_amd64_gcc493 -r "comp=comp" -R comp@<LATESTTAG> -t MYDEV -s "prep sw post" $PWD dqmgui/bare  #for the latest tag, have a look here: https://github.com/dmwm/deployment/releases
     source current/apps/dqmgui/128/etc/profile.d/env.sh
     </verbatim>

%ENDTWISTY%

---+++ Add new plots to the !QuickCollection

%TWISTY{mode="div" showlink="Add new plots to the !QuickCollection" hidelink=" Hide 'Add new plots to the !QuickCollection'" remember="off" showimgleft="%ICONURLPATH{toggleopen-small}%" hideimgleft="%ICONURLPATH{toggleclose-small}%" class="twikiHelp"}%

To add a plot in the quick collection:
   1 Open the ==workspace-<flavor>.py== you want to update
   1 Add the !QuickCollection path (get inspiration from other lines). Please always start by adding plots in layouts before putting them in !Quickcollection. The plots in !QuickCollection should *always* be in layouts, if possible with the same index.

%ICON{help}% Plots in !QuickCollection are the most important plots a shifter will have to look at. They need to be properly documented with a good description and a specific description in the related twiki:
   * ==for !SiStrip== : [[https://twiki.cern.ch/twiki/bin/viewauth/CMS/SiStripOfflineDQMHistoDescription]]
   * ==for Pixel== : [[https://twiki.cern.ch/twiki/bin/view/CMS/PixelOfflineDQMHistoDescription]]
   * ==for Tracking== : [[https://twiki.cern.ch/twiki/bin/view/CMS/TrackingOfflineDQMHistoDescription]]

%ENDTWISTY%

---+++ Edit the layouts (add new plots, add a reference, add a description...)

%TWISTY{mode="div" showlink="Edit the layouts (add new plots, add a reference, add a description...)" hidelink=" Hide 'Edit the layouts (add new plots, add a reference, add a description...)'" remember="off" showimgleft="%ICONURLPATH{toggleopen-small}%" hideimgleft="%ICONURLPATH{toggleclose-small}%" class="twikiHelp"}%

This section explains how to play with the layouts in the dqmgui/layouts folder ( [[https://github.com/dmwm/deployment/tree/master/dqmgui/layouts][github link]]).

---++++!! The folder structure

There are different python files for each workspaces and flavors of the !DQM !GUI. They follow this nomenclature: ==T0 = offline; shift = shift workspace; overview = feedback for collisions=. %BR%
Here is the list of interest for the tracker community:%BR%

For !SiStrip:%BR%
   * sistrip-layouts.py for online flavor of the GUI
   * sistrip_T0_layouts.py for the offline GUI
   * shift_sistrip_layout.py for what is in the Shift/SiStrip workspace of the online GUI
   * shift_sistrip_T0_layout.py for what is in the Shift/SiStrip workspace of the offline GUI
   * sistrip_overview_layouts.py for the layouts of the tracking feedback collection in both offline and online GUI
   * sistriplas-layout.py for the sistripLAS workspace of the online GUI (not used)

For Pixel:%BR%
   * pixel-layouts.py for online flavor of the GUI
   * pixellumi-layouts.py for the layouts of the workspace of !PixelLumi... which only exist in "Everything" (not used)
   * pixel_T0_layouts.py for offline flavor of the GUI
   * shift_beampixel_layout.py for what is in the Shift/BeamPixel workspace of the online GU
   * shift_pixel_layout.py for online flavor of the GUI
   * shift_pixellumi_layout.py for what is in the Shift/PixelLumi workspace of the online GU
   * shift_pixel_T0_layout.py for what is in the Shift/Pixel workspace of the offline GUI

For Tracking:%BR%
   * shift_tracking_T0_layout.py for what is in the Shift/Tracking workspace of the offline GUI
   * tracking_T0_layouts.py for the tracking worskpace in the offline GUI

Note: You can see at the top of the file where is the layout folder located in case you have any doubts.

---++++!! The python files structure

At the top of each python file, you can find the information of where is this layout applied, then follows the definition of each layout with:
   * a title for the "layout object" (a "layout object" can contain several plots)
   * then for each plot in this object, you have the following options:
      * ==path==, the path in !DQM !GUI of this histogram
      * ==description==, the description that will be shown when the "Describe" button is hit in the !GUI. %BR%%ICON{help}% It is %RED%*mandatory*%ENDCOLOR% to add a description to new plots. If you created a new plot and want to add it to the layouts, it's *your* role to provide a good description to it. Furtheremore, if the plot is added to the layouts, then its description should also figure [[%SCRIPTURL{"view"}%auth/CMS/SiStripOfflineDQMHistoDescription][here]].
      * ==reference==, to automatically plot the reference on this plot. Most of the time this option is set to "no".

Therefore, to add a new layout, just create a new layout object. You can get inspiration by looking at the other lines in the code.%BR%
%ICON{help}% Always test your code, even if it looks trivial, a typo is really easy to make and can be deadly to the GUI.


---++++!! Update the documentation

Plots in Layouts are (after the !QuickCollection ones) the most important plots a shifter will have to look at. They need to be properly documented with a good description and a specific description in the related twiki:
   * ==for !SiStrip== : [[https://twiki.cern.ch/twiki/bin/viewauth/CMS/SiStripOfflineDQMHistoDescription]]
   * ==for Pixel== : [[https://twiki.cern.ch/twiki/bin/view/CMS/PixelOfflineDQMHistoDescription]]
   * ==for Tracking== : [[https://twiki.cern.ch/twiki/bin/view/CMS/TrackingOfflineDQMHistoDescription]]

If you have doubts about what to write there, contact the person who asked you to make this plot.



%ENDTWISTY%

---+++ Edit the style (log scale, colz option, lines, etc.)

%TWISTY{mode="div" showlink="Edit the style of some plots in the GUI (log scale, colz option, etc.)" hidelink=" Hide 'Edit the style of some plots in the GUI (log scale, colz option, etc.)'" remember="off" showimgleft="%ICONURLPATH{toggleopen-small}%" hideimgleft="%ICONURLPATH{toggleclose-small}%" class="twikiHelp"}%

This section explains how to play with the style of the plots, in the dqmgui/style folder ( [[https://github.com/dmwm/deployment/tree/master/dqmgui/style][github link]])

---++++!! The folder structure

Here, there are less files, since those C++ files are both for online and offline flavor of the GUI.

For !SiStrip:
   * ==SiStripRenderPlugin.cc==, for the !SiStrip workspace
   * ==SiStripLASRenderPlugin.cc==, for the !SiStripLAS workspace *(not used)*

For Pixel:
   * ==SiPixelRenderPlugin.cc==, for the !Pixel workspace
   * ==BeamPixelRenderPlugin.cc==, for the !BeamPixel workspace

---++++!! The C++ files structure
   1 The first function (==applies==) shows in which folders this !RenderPlugin is applied. One has to be careful to do not alter other folders (like !BeamPixel with Pixel) with too evasive keywords.
   1 Then there are several ==preDraw== functions for each kind of histogram (!TH2F, !TH1F, ...). This is where you should add your customization for new plots, such as *colz option*. %BR %ICON{help}% A common mistake is to add the plot in the wrong category (e.g. !TH2F instead of a !TProfile2D). Therefore be careful and *test your changes*!
   1 Then there are ==postDraw== functions. This is where you add things that has to come after drawing the histograms, like adding an "error message" when something happens, or to change to a logarithmic axis if there are more than N entries, etc. This is also where you can add lines to be drawn on the plot to show expected ranges.

As always, what you want to implement is probably already implemented for an other plot, so get inspiration from there! %BR%
%ICON{help}% When adding a new plot (through the ==name.find== function), please be careful to:
   * avoid duplicating functions. Mayeb there is already a line that is changing your plot, but it just misses the (e.g) colz option.
   * be smart, don't create 5 lines if you have 5 new plots with almost the same name and that requires the same options.

%ENDTWISTY%

#TestChanges
---+++ Test your changes

%TWISTY{mode="div" showlink="Test your changes" hidelink=" Hide 'Test your changes'" remember="off" showimgleft="%ICONURLPATH{toggleopen-small}%" hideimgleft="%ICONURLPATH{toggleclose-small}%" class="twikiHelp"}%

Testing your changes is the most important step. Please be sure to *always* check what you did before sending any pull request.

   1 You should already have a !DQM instance running if you followed the two first steps of this [[#HowToList][How to]] section. If not, please refer to those two steps first.
   1 Then, re-deploy to copy your changes to your running GUI:
     <verbatim>
     cd /tmp/$USER/testGui
     $PWD/deployment/Deploy -A slc6_amd64_gcc493 -r "comp=comp" -R comp@<LATESTTAG> -t MYDEV -s "prep sw post" $PWD dqmgui/bare  #for the latest tag, have a look here: https://github.com/dmwm/deployment/releases
     source current/apps/dqmgui/128/etc/profile.d/env.sh
     </verbatim>
   1 And restart your !GUI (depending on the flavour you're working on):
      <verbatim>
      $PWD/current/config/dqmgui/manage -f dev restart "I did read documentation"
      $PWD/current/config/dqmgui/manage -f online restart "I did read documentation"
      $PWD/current/config/dqmgui/manage -f offline restart "I did read documentation"
      $PWD/current/config/dqmgui/manage -f relval restart "I did read documentation"
      </verbatim>
   1 Have a (good) look at your changes:
      <verbatim>
      hostname:8060/dqm/dev
      hostname:8070/dqm/online-dev
      hostname:8080/dqm/offline
      hostname:8081/dqm/relval
      </verbatim>

%ENDTWISTY%

---+++ Send a Pull Request and close your GUI

%TWISTY{mode="div" showlink="Send a Pull Request and close your GUI" hidelink=" Hide 'Send a Pull Request and close your GUI'" remember="off" showimgleft="%ICONURLPATH{toggleopen-small}%" hideimgleft="%ICONURLPATH{toggleclose-small}%" class="twikiHelp"}%

---++++!! Make the Pull Request

   * Make sure you have forked the _deployment_ repo to your account. You can do this [[https://github.com/dmwm/deployment][here]] by clicking on the _Fork_ button.
   * Create a separate branch in git (strongly recommended).
     <verbatim>cd /tmp/$USER/testGui/deployment</verbatim>
     <verbatim>git checkout -b MyNewBranchName</verbatim>
   * Commit your changes. Please use a smart and logical message.
     <verbatim>git status</verbatim>
     <verbatim>git add modifiedFile1 modifiedFile2 ...</verbatim>
     <verbatim>git commit -m "smart and logical mesage"</verbatim>
   * Add your private repo as a remote and push the branch to your private repo.
     <verbatim>git remote add my-deployment git@github.com:<my_github_account>/deployment.git</verbatim>
     <verbatim>git push my-deployment <my_branch_name></verbatim>
   * Before making any further request, make sure that you have thoroughly [[#TestChanges][tested]] your development!
      * Restart the !DQM GUI and make sure your modified layouts and render plugins compile without any problems.
      * Please keep your code clean: Make sure you have no trailing spaces or trailing newlines.
      * If you need to modify things, you can simply make commits to your current branch and push it again.
   * If you are changing anything to the *shift layouts* for your subsystem: *update your subsystem's shift instructions*
      * Please understand that out of respect for our shifters, we will not even consider your change if it's not documented first.
      * Off course, the instructions should not appear for the shifter until the updated layout is on production. For that reason, please place the new instructions between =&lt;!--= and =--&gt;= in the Twiki source, so that we can review them, without the shifter seeing them.
      * If you have questions, don't hesitate to contact us.
   * After successful testing (and, if applicable, update of the instructions), go to Github and create a pull request for the branch you just pushed. Please give the pull request a smart and logical title and description.
   * We will see that you created a pull request, but please send an email to cms-dqm-coreTeam@cern.ch nonetheless.
      * Let us know if this is a patch for the Online !DQM GUI, the Offline !DQM GUI, the Relval !DQM GUI or all of them.
      * Mention which pull request you're emailing about.
      * Don't hesitate to mention any other useful information.
   * We will pick this up, test if the patch process works and install it on the _Online Test_ system and the _Online Playback_ system.
      * If the modification can be tested on the Online flavor, you can test it now.
         * Depending on what you want to test, we have to !DQM GUI test servers looking at different data:
            * On the _Online Test_ system, you will find exactly the same data as on the production system.
            * On the _Online Playback_ system, you will find data produced with the latest CMSSW developments that are currently being tested.
         * If testing is successful, we will propagate it to the !DQM GUI Online Production systems.
         * After that we will also accept the pull request so that it will be merged in the next official release (for Offline and Relval).
      * If the modification can only be tested on the Offline or Relval flavor:
         * We will accept the pull request so that it can be tested during the testing period of the next official release.
         * You test during the official testing period and let us know if everything works as expected.

---++++!! Close your !DQM !GUI (depending of the flavor)

This is a very important step do not overload the lxplus machine for nothing.

<verbatim>
   $PWD/current/config/dqmgui/manage -f dev stop "I did read documentation"
   $PWD/current/config/dqmgui/manage -f online stop "I did read documentation"
   $PWD/current/config/dqmgui/manage -f offline stop "I did read documentation"
   $PWD/current/config/dqmgui/manage -f relval stop "I did read documentation"
   </verbatim>

%ENDTWISTY%

#AdditionalInformation
---++ 4. Additional information

In this section, we present tutorials for experts or for more technical issues (like tunneling).

#TunnelingLxplus
---+++ Tunneling to lxplus

%TWISTY{mode="div" showlink="Tunneling to lxplus" hidelink=" Hide 'Tunneling to lxplus'" remember="off" showimgleft="%ICONURLPATH{toggleopen-small}%" hideimgleft="%ICONURLPATH{toggleclose-small}%" class="twikiHelp"}%
To ease things, we strongly recommend to use a browser add-on to play with proxy settings. In this section we present a way to do it on firefox with the add-on ==FoxyProxy==

   1 Install the add-on
      * Open the add-on page on firefox, for this write ==about:addons== in the address bar.
      * Search for "FoxyProxy" and install the "FoxyProxy Standard" version of this add-on.
      * Restart your browser.
   1 Configure your proxy
      * Once installed, you will see a "fox head" icon on the right of your address bar.
      * Right-click on it and click on option.
      * Then click on "Add new proxy":
      * select "Manual Proxy Configuration"
      * Host or IP Address: localhost
      * Port: 11080
      * Select SOCKS Proxy
      * Switch tab and go in "General" : there you can choose your settings name (set it to lxplus for example)
      * Switch tab and go in "URL patters": Click on "add New Pattern"
      * choose a pattern name (for example lxplus)
      * in URL pattern, set expression to tell when this proxy should be active: set it to <b> ==lxplus== </b> and click on OK.
      * to avoid bugs in some cases, add again a new pattern. Name it "NOT google" and set the URL pattern to <b> ==google== </b>. But this time select the blacklist option and then press OK.
      * eventually, since tunneling to lxplus is also used when working on vocms machine, add a new pattern. Name it "vocms" and set the URL pattern to <b> ==vocms== </b> and then press OK.
      * then close the windows and right-click again on the "fox head" icon and select "Use proxies based on their pre-defined patterns and priorities"
   1 Start the tunneling
      * Once !FoxyProxy set, you won't have to touch it anymore. When you will connect to lxplus or vocms it will automatically try to use your tunnel options. All you have to do is to open the tunnel, with the following command:
        <verbatim>
        ssh -D 11080 <username>@lxplus.cern.ch
        </verbatim>
And you can now navigate through your personal GUI, or the vocms machines, ...
 %ENDTWISTY%

#PlayOwnGui
---+++ Play with our own online GUI (experts only)

%TWISTY{mode="div" showlink="Create our own online GUI (expert only)" hidelink=" Hide 'Create our own online GUI (expert only)'" remember="off" showimgleft="%ICONURLPATH{toggleopen-small}%" hideimgleft="%ICONURLPATH{toggleclose-small}%" class="twikiHelp"}%
In order to be able to reproduce the way the data is fed in the online flavor or the !DQM !GUI, a dedicated machine has been set up.
Here are the different steps to be able to connect to this machine and use it. If one wants to know more about how to set up such a machine, please refer to the [[#CreateOurOwnGUI][Create our own GUI]].
Also, please note that the central DQM has made its own general twiki, which englobe this one : [[CMS.DQMOnlineTesting][DQMOnlineTesting]].

   1 Working on the right machine
      <div style="margin-left: 2em;">
      Tunnel to lxplus, then to cmsuser, then connect to the dedicated machine for tracker (fu-c2f11-21-01).
      <verbatim>
      ssh -tC username@lxplus.cern.ch ssh -tC username@cmsusr.cern.ch ssh -C username@fu-c2f11-21-01.cms
      </verbatim>
      </div>
   1 Set up CMSSW and the proxy (just do it the first time)
       <div style="margin-left: 2em;">
      On the test machine, edit your .bashrc and add the following
      <verbatim>
      prod_machines=fu-c2f11-21-01
      localhost=$(hostname)
      
      case "${prod_machines[@]}" in
      *$localhost*)
          # Getting cmssw
          source /opt/offline/cmsset_default.sh
          export SCRAM_ARCH=slc6_amd64_gcc491
          # Proxy settings
          export http_proxy="http://cmsproxy.cms:3128"
          export https_proxy="https://cmsproxy.cms:3128/"
          export NO_PROXY=".cms"
          ;;
      esac
      </verbatim>
      </div>
   1 Set up github (just do it the first time)
      <div style="margin-left: 2em;">
      Follow the usual steps to set up [[http://cms-sw.github.io/faq.html#how-do-i-subscribe-to-github][github]]:
      <verbatim>
      git config --global user.name <name> <last name>                 
      git config --global user.email <email adress>
      git config --global user.github <username>
      </verbatim>
      [[https://help.github.com/articles/generating-ssh-keys/][Register your ssh key]].<br />
      However, you will need to follow some additional steps (those instructions originate from the [[%SCRIPTURL{"view"}%auth/CMS/DQMOnline#How_to_setup_GIT_to_work_from_in][DQMOnline]] twiki. Please note this guide only works if github access is via ssh (--ssh option)) <ul> <li>Login to cmsusr and edit your .ssh/config file adding:
        <verbatim>
        Host github.com
            User git
            ProxyCommand ssh cmsusr.cms nc %h %p
        </verbatim> </li> <li>remove the folder =~/.cmsgit-cache/= </li> <li>copy your =.ssh= folder contents from lxplus to cmsusr</li> <li>Make sure that you set the ssh settings for 'forward agent' is =true= </li> <li>login to the machine you want to reach (here fu-c2f11-21-01)</li> <li>type =kinit= and insert the password</li> <li>from inside any project, after =cmsenv= git and the cms-git-tools should work with the --ssh option
      </li> </ul></div>
   1 CMSSW and addition of DQM/Integration
      <div style="margin-left: 2em;">
      On the machine fu-c2f11-21-01, write<br />
      <verbatim>
      kinit <username>
      scram list
      scram project -n <CMSSW_version>_official <CMSSW_version>
      cd <CMSSW_version>/src
      cmsenv
      git-cms-addpkg DQM/Integration --ssh 2>&1 | cat;  #It is possible that a lot of "killed by signal 1" messages appear and that you will have to enter your password a lot of time. But this is not an issue.
      scram buid -j8
      </verbatim>
      Do a quick check:
      <verbatim>
      cat /etc/dqm_run_config
      </verbatim>
      This should give the following output:
      <verbatim>
      [host]
      type = userarea
      </verbatim>
      </div>
   1 Online configuration
      <div style="margin-left: 2em;">
      Now that we have the DQM/Integration package, we can configure two things: <ul> <li>The source of the input:<br />
         in DQM/Integration/python/config/inputsource_cfi.py, under 'runInputDir', change ='/tmp'= by ='/fff/BU0/output/lookarea'= </li> <li>The destination of the output:<br />
         in DQM/Integration/python/config/environment_cfi.py, change userarea ='collectorHost'= by ='fu-c2f11-21-01'= <br />
      Then, don't forget to build the project:
      <verbatim>
      scram b -j8
      </verbatim>
      </li> </ul></div>
   1 Installing an Online DQM GUI (already done)
      <div style="margin-left: 2em;">
      This step shouldn't be needed since one GUI have been set once for all. However, if the expert user would need to set up a whole new GUI, some information are given in [[#CreateOurOwnGUI][installing an Online DQM GUI]].
      </div>
   1 Feed the GUI
      <div style="margin-left: 2em;">
      Always start to check if the run you want to look is available:
      <verbatim>
      ls -l /fff/BU0/output/lookarea/runXXXXXX/
      </verbatim>
      If you need collisions, check it in the WBM Run Summary.<br /><br />
      Create the =./upload= directory fom where you will launch =cmsRun=
      <verbatim>
      cd DQM/Integration/python/clients/
      mkdir ./upload
      </verbatim>
      Run the typical configuration of your subsystem:
      <verbatim>
      cmsRun subsystem_config.py runNumber=abcxyz
      #E.g cmsRun sistrip_dqm_sourceclient-live_cfg.py runNumber=261259
      </verbatim>
      This step should take a few minutes. It will "freeze" automatically after the last !LS is reached.
      </div>
   1 Check the DQM GUI
      <div style="margin-left: 2em;">
      Tunnel to P5 from outside:
      <verbatim>
      ssh -tCXgL 22223:localhost:22224 username@lxplus.cern.ch ssh -tCXgL 22224:localhost:22225 username@cmsusr.cern.ch ssh -CXgD 22225 username@fu-c2f11-21-01.cms
      </verbatim>
      Now you can use your browser with port 22223.<br />
      The address is: fu-c2f11-21-01:8070/dqm/online-dev
      </div>

%ENDTWISTY%

#CreateOurOwnGUI
---+++ Installing our own DQM GUI (experts only)

%TWISTY{mode="div" showlink="Installing our own DQM GUI (experts only)" hidelink=" Hide 'Installing our own DQM GUI (experts only)'" remember="off" showimgleft="%ICONURLPATH{toggleopen-small}%" hideimgleft="%ICONURLPATH{toggleclose-small}%" class="twikiHelp"}%

This section is still to be done. If needed, please contact hugo.ruben.delannoy@cern.ch.
In any case you will need special rights to do this.

%ENDTWISTY%
Topic revision: r1 - 2017-05-11 - HugoRubenDelannoy
Account
- Log In
- Cern Search
- TWiki Search
- Google Search
Sandbox All webs
Edit
Attach
Copyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback