SCRAM for developers of CMSSW

Contents

• Introduction
• SCRAM
• SCRAM commands
  • Searching for available releases
  • Creating a developer area based on release area
  • Compiling/Building code
    • scram build command-line flags
    • scram build command-line targets
    • Running test executables generated from test directory
    • Tips/Tricks
    • Errors/Warnings
      • Missing Packages/Tools
      • Seal Plugins
      • LCG Dictionaries
      • "scram b" did not re-build any product after changing an external tool
      • Others
  • Getting list of external tools (and their versions) setup for your dev area
  • Getting details of an external tool
• Building debug shared libraires for a non-debug SCRAM ARCH
• Skipping shared library checking tests during the compilation of your code
• Renaming/Moving a SCRAM-based project's dev area
• Compiling against a different version of external tool
• Missing/Adding library for an external tool
• Adding new external tools
• Changing an external tool description

Introduction

SCRAM

SCRAM commands

• search SCRAM about available SCRAM-based project for a given architecture
• create a developer area based on a given release where a developer can compile/build/test his/her code
• compile/build code
• get help on different scram commands

Searching for available releases

arch

"scram arch" is the command which tells you what architecture you are working on. You can force a architecture to be a different by setting the SCRAM_ARCH environment variable. e.g. on a SLC3, 32 bit machine with GCC compiler version 3.2.3 the "scram arch" command will print something like slc3_ia32_gcc323. Now if one wants to work on a debug version (if available) then he/she can set the SCRAM_ARCH environment variable to slc3_ia32_gcc323_dbg. e.g. doing something like this csh/tcsh shell

In csh/tcsh:
> setenv SCRAM_ARCH slc3_ia32_gcc323_dbg
OR In bash/sh:
> SCRAM_ARCH=slc3_ia32_gcc323_dbg; export SCRAM_ARCH

will force the scram architecture to slc3_ia32_gcc323_dbg unless one explicitly forces the scram arch using -arch scram-arch command line argument.

list

"scram list ProjectName" is the command which shows which releases are available for a given scram architecture. e.g. doing something like

> scram list CMSSW

will print the list of all available CMSSW releases based on current scram arch. OR

> scram --arch slc4_ia32_gcc345 list CMSSW

will show the list of all CMSSW projects released for slc4_ia32_gcc345. e.g.

> scram list CMSSW

will print something like this

Listing installed projects....

--------------------------------------------------------------------------------
| Project Name  | Project Version          | Project Location                  |
--------------------------------------------------------------------------------
  CMSSW           CMSSW_x_y_y                
                                         --> /afs/cern.ch/cms/sw/slc3_ia32_gcc323/cms/cmssw/CMSSW_x_y_z
  CMSSW           CMSSW_a_b_c_pre1           
                                         --> /afs/cern.ch/cms/sw/slc3_ia32_gcc323/cms/cmssw/CMSSW_a_b_c_pre1

help

The one very useful scram command is "scram scram-command -help" which provides online scram help on different scram commands. e.g. "scram list -help", "scram arch -help" etc.

Creating a developer area based on release area

project

"scram project ProjectName ProjectVersion" is the command to do this. e.g. doing

> scram project CMSSW CMSSW_x_y_z_pre1

will create a CMSSW_x_y_z_pre1 developer's area in your current working directory. The next step is to check out the code (on which you want to work) from github in to the src directory of newly created dev area.

> cd CMSSW_x_y_z_pre1/src
> cmsenv
> git cms-addpkg Subsystem/Package

Please remember that if you want to create a new Subsystem/Package in CMSSW repository then please use the CMS tag collector to generate a request. Details about the sub-directories under Subsystem/Package are available here.

Compiling/Building code

build

This command "scram b" will build every thing available under your current working directory. e.g. running it from your dev's src directory (e.g. CMSSW_x_y_z_pre1/src) will build every Subsystem/Package available and running it from a specific Subsystem/Package directory (e.g. CMSSW_x_y_z_pre1/src/subsystemA/packageB) will only build this package and all the other packages it depends on (if needed). Please remember that once a Subsystem/Package has been built then changing

• src/*.cc files should only re-compile those src/*.cc files and re-link the shared library
• interface/*.h files should only re-compile all those src/*.cc files which depend on those headers and re-link the shared library.
• BuildFile file should re-compile all the src/*.cc files and re-link the library and also re-build all other Subsystem/Packages which depends on this package. e.g. if subsystemA/packageC and subsystemA/packageD depends on subsystemA/packageB then changing the subsystemA/packageB/BuildFile will not only rebuild subsystemA/packageB but also subsystemA/packageC and subsystemA/packageD.

Command-line flags for scram b

There are some useful command-line arguments which you can pass to the "scram b" command. e.g. following are the command-line arguments which scram passes to the gmake (the actual building tool)

• -k: Normally "scram b" command stops its processing at first compile/build error. If you want to keep on running then add this command-line argument e.g. "scram b -k".
• -j n: You can use -j n (where n is an integer) to run n simultaneous processes for compilation/building. e.g. "scram b -j 4" will run 4 simultaneous processes to compile/build.
  NOTE: This command-line option will work properly only for CMSSW nightly builds, CMSSW_1_3_0_preX and above.

Following are the command-line arguments which are used directly by scram for building your code.

• -f: Adding this command-line argument tells scram to skip the checking of the configuration files and BuildFiles changes and go directly to the building. It is only useful when you have already build a package and now you have only changed any *.h or *.cc file.
  NOTE: If you have either checked out a new package from git or deleted a existing package or modified the BuildFile of already built package then please do not use this command-line argument. See tips/tricks for more detail.
• -r: Adding this command-line argument tells scram to re-build the project cache (i.e. by reading all the BuildFiles and updating the internal cache) before running any gmake commands.
  NOTE: For CMSSW nightly builds, CMSSW_1_3_0_preX and above, you do not need to use this command-line argument unless you have changed any .tmpl file in your project's config directory.
• -v: By default in CMSSW, scram b prints information about what the build system is doing not how it is doing. Adding this command-line option also prints the commands used for building. Please note that if you already have run scram b (i.e. without -v option) then running scram b -v next time will have no effect (unless the internal cache was updated for some reason). The alternative to this is SCRAM_BUILDVERBOSE environment variable. Setting this environment variable, to a non-empty value, will always run scram build in verbose mode.

Command-line targets for scram b

There are some special command-line argument (actually the gmake targets) available which are good to know. e.g.

• clean: "scram b clean" will remove already build products (libraries/binraries/scripts) and the tmp area used to build those e.g. the *.o files etc. Running this form a subsystem/package will remove all the products generated from this package and running it from a subsystem directory will remove all the products generated from all the packages of this sub-system.
• echo_xyz: "scram b echo_xyz" (where xyz is an internal gmake variable e.g. CXX, CPPDEFINES etc.) will print the value of xyz gmake variable.
• ToolUpdated_toolname: "scram b ToolUpdated_toolname" (where toolname is an external tool setup for your project e.g. qt, boost, root, rootcore etc.) will update the internal build cache for that tool so that when next time you do a "scram b" then all the products, which depend on this tool, could re-compile/re-build properly using new path of that tool. You need to run this after running any "scram setup ...." command.
  NOTE: This target is only available in CMSSW nightly builds, CMSSW_1_3_0_preX and above.
• ExternalLinks: "scram b ExternalLinks" will update all the symlinks for all the tools setup locally in your developement area. You only need to run this if you had done "scram setup".
• subsystem: "scram b subsystem" (where subsystem is the name of subsystem you want to build e.g. DataFormats, FWCore etc.) will build every thing for that subsystem.
• subsystem_clean: "scram b subsystem_clean" (where subsystem is the name of subsystem you want to clean e.g. DataFormats, FWCore etc.) will remove all the product generated from the subsystem and also clean the tmp area for that subsystem.
• prodname: "scram b prodname" (where prodname is the name of the product(lib/bin) you want to build e.g. for a library generated from subsystemA/PackageB/src directory the prodname will be subsystemAPackageB
  NOTE: This target is only available in CMSSW nightly builds, CMSSW_1_3_0_preX and above.
• prodname_clean: "scram b prodname_clean" (where prodname is the name of the product(lib/bin)) will remove the product prodname and also clean all the tmp files for that product.
  NOTE: This target is only available in CMSSW nightly builds, CMSSW_1_3_0_preX and above.
• subsystem/package: "scram b subsystem/package" (where subsystem/package is the package you want to build e.g. FWCore/Framework etc.) will build every thing for that subsystem/package.

Running test executables generated from test directory

If you have generated some test executables/binaries from your test directory then by running "scram b runtests" will run all the tests and pass the required command-line arguments (defined in the test/BuildFile) to each test. Please see test runner flag for more details.

TIPS and TRICKS

• Check out only the packages which you want to edit/modify. e.g. if you want to edit/modify a subsystemA/packageB then only checkout subsystemA/packageB and not the whole subsystemA. Less packages in your developer area means less time to compile/build and more time for testing.
• -f: Make use of -f option, it will save you some compilation time. Please do not use this flag if you have
  • freshly checked out a new package
  • removed an already check out package
  • modified or added a CMS.BuildFile
  • added a new package's sub-directory e.g. data, test, bin, scripts, python, plugins
  • added Qt Q_OBJECT macro in any *.h file, so that the Qt moc should run for this
  • added new files for
    • Root dictionaries: src/*LinkDef.h
    • LCG dictionaries: src/classes.h and src/classes_def.xml
    • Code generation: src/*desc.xml
    • Flex/Bison: src/*lex.l and src/*parse.y

It is safe to use -f option if you have modified/added *.h, *.cc files.

• -j: Make use of -j N option (where N is ~= twice the number of processors available). When you have multiple packages in your development area then its hard to read the output of scram b -j n. If you set BUILD_LOG environment variable to yes then doing scram b will print the build log for each package separately. e.g. one can do
  • for tcsh/csh: env BUILD_LOG=yes scram build
  • for bash/sh: BUILD_LOG=yes scram build
• You do not need to use -r flag ever. The only time you need to use this option is when you have changed any .tmpl file in your project's config directory.
• Try not creating any log file under your project's src tree (any where under project's src directory and its sub-directories). e.g. running your test executable and re-directing the output to some log file in your project's src directory tree is not good. Because creating/removing a file under your project's src directory will change the timestamp of that directory and this means that next "scram b" command is going to update your project cache (unless there is -f flags used), so next compilation will take a bit more time.

Getting list of external tools (and their versions) setup for your dev area

Getting details of an external tool

Building debug shared libraires for a non-debug SCRAM ARCH

<Flags CXXFLAGS="-g">

• top level BuildFile.xml (i.e. config/BuildFile.xml) to generate all libraries/executables from this dev area with debugging information OR
• in specific BuildFile.xml (i.e. src/Subsystem/Package/BuildFile.xml, src/Subsystem/Package/test/BuildFile.xml, src/Subsystem/Package/bin/BuildFile.xml etc.) to generate all libraries/executables from this specific package with debugging information.

• scram b clean; scram b USER_CXXFLAGS="-g"

   ....
   .../external/boost/1.33.1=BOOST
   .../external/boost/1.33.1=BOOST_PYTHON
   .../external/bz2lib/1.0.2=BZ2LIB
   .../external/castor/2.1.1-4=CASTOR
   ...
   .../lcg/root/5.13.04e=ROOTCORE
   .../lcg/root/5.13.04e=ROOTRFLX

<Use name=toolname>
<Use lib=newlib>