Instructions for how to install and run the HEP-SPEC06 (HS06) benchmarking suite

This page documents how to install and run the HS06 benchmarking suite in order to determine the HS06 score of your machine.

A typical use case is the following: you run an algorithm on machine A and would like to compare the CPU performance/timing against another (or the same) algorithm whose performance was previously measured on machine B. The HS06 score provides a conversion factor to directly compare timing measurements obtained on different machines:

algorithm runtime on machine X (in seconds) x HS06 score for machine X = algorithm runtime in HS06-seconds

You can find an in-depth description of how to run the HS06 benchmarks on the HEPiX page. This tutorial summarizes some of the information contained in the HEPix page and additionally provides instructions on how to manually install the HS06 suite on a modern system.

Disclaimers

The following instructions have been tested on a system with two 2.3 GHz Intel Xeon E5-2698 v3 (Haswell) CPUs with 32 total (physical) cores and 2 threads/core running CentOS7.

Note that HS06 is based on the SPEC CPU2006 benchmark suite which has been retired as of 2018. The latest version of the benchmark suite is v1.2, released in September 2011. In particular, this means that several patches have to be applied to the HS06 source code in order to get the benchmarks to compile/run on a more modern system. As such, please keep in mind that your experience may vary if you try running these instructions on a different system -- in this case, Google will be your friend in trying to fix any compilation issues you might encounter.

1. Installing the HS06 suite

Prerequisites

Before following along with the rest of this tutorial, please check that:

• your system satisfies the following:
  • operating system: RHEL or any of its clones (CentOS, Scientific Linux, Scientific Linux CERN)
  • GNU Compiler Collection (GCC) available
• you have access to the following source files:
  • an archive of the SPEC CPU2006 benchmark suite v1.2 (this represents the SPEC part of HEP-SPEC06). Since this a licensed product, a public link can't be provided here. Try asking a colleague/supervisor if they can provide you with a copy. For the instructions presented in this tutorial I will assume this archive is called speccpu2006-1.2.tar.bz2.
  • some configuration files (these represent the HEP part of HEP-SPEC06) -- the configuration files are available as a gzipped tar archive directly from HEPiX, which I will link here: spec2k6-2.24.tar.gz

Installation procedure

This is a brief reminder that the following instructions have only been formally tested on CentOS7.

The first step consists of going to the directory where we want to install the benchmark suite and unpacking the source files. Here we'll assume the archives have already been copied to this installation directory.

cd $MYINSTALLDIR
tar -xf spec2k6-2.24.tar.gz  # configuration files are *_cern.cfg
tar -xf speccpu2006-1.2.tar.bz2 # this extracts the source files to the speccpu2006-1.2 directory
# the name of the speccpu2006-1.2 directory might differ depending on where you get the archive from
# we will sometimes generically refer to this as $SPECDIR

If you look inside the benchmark directory $SPECDIR you will see that it contains an install script, install.sh. This used to be the recommended way to install the suite back when it was first released and for several years after that -- the install script essentially automates many of the steps described below.

Unfortunately, using the install script doesn't seem to work reliably/anymore on more modern operating systems, so we will instead have to manually build the SPEC CPU2006 tools suite (later on in the tutorial, the SPEC specific tools will be used to build the benchmarking executables).

Optional: If interested, you can find more information about building the SPEC tools suite here.

The source files for the SPEC tools live in the $SPECDIR/tools/src/ directory. Recal that for this tutorial $SPECDIR=speccpu2006-1.2, so we will go there with cd speccpu2006-1.2/tools/src/ . Here is a list of all the folders you should find in $SPECDIR/tools/src/:

Algorithm-Diff-1.1902  HTML-Tagset-3.20  String-ShellQuote-1.04     XML-SAX-0.96          buildtools.log     makedepf90-2.8.8  specinvoke
File-NFSLock-1.20      IO-stringy-2.110  Text-CSV_XS-0.79           XML-SAX-ExpatXS-1.31  dmake-4.11         ntzsh             specsum
Font-AFM-1.20          MIME-tools-5.428  TimeDate-1.20              bin                   expat-2.0.1        perl-5.12.3       tar-1.25
GD-1.19                MailTools-2.04    URI-1.56                   buildtools            libwww-perl-5.837  redistributable   xz-5.0.0
HTML-Parser-3.68       PDF-API2-0.73     XML-NamespaceSupport-1.11  buildtools.bat        make-3.82          rxp-1.5.0

Before we can build the tools, we will need to patch some of the source files (as mentioned in the Disclaimers) -- otherwise we will encounter various compilation errors. The following set of patches has been determined mostly by googling the various compilation errors and testing out the proposed solutions. Again, I want to point out that your experience may vary and more/different patches may be necessary if you want to get HS06 running on a different system (and especially if you are using a different version of SPEC CPU2006)!

Patch #1:

In the file specsum/tests/test-symlink.h comment out line 70:

ASSERT (errno == EEXIST || errno == EINVAL);

Patch #2:

In the file TimeDate-1.20/t/getdate.t replace line 159:

# my $offset = Time::Local::timegm(0,0,0,1,0,70); # original line
my $offset = Time::Local::timegm(0,0,0,1,0,1970); # new line

Patch #3:

In the file tar-1.25/gnu/stdio.in.h comment out line 147:

_GL_WARN_ON_USE (gets, "gets is a security hole - use fgets instead");

Patch #4:

In the file specsum/gnulib/stdio.in.h comment out line 162:

_GL_WARN_ON_USE (gets, "gets is a security hole - use fgets instead");

Now we can go ahead and build the tools with the following command:

cd speccpu2006-1.2/tools/src/ # make sure we are in tools/src/
./buildtools

This will start the build process and will take quite a bit of time to finish, depending on your system. At some point during the build process you will see a message that some perl tests have failed (something on the order of a few out of ~1000 tests). It should be okay to just enter y at the prompt (meaning ignore the failed tests, which are probably due to the perl source code being quite old) and hit enter to continue with the build. After some more time passes the build process should finish successfully.

There are some extra tests that we can run at this point to double-check that the tools were successfully built:

cd speccpu2006-1.2 # this is our $SPECDIR
. ./shrc # source the env
which runspec # this should print $SPECDIR/bin/runspec
# ask the major tools to identify themselves
runspec -V
# more comprehensive tests on specperl
runspec --test

The output of runspec -V will contain, among other things, a list of the spec tools:

Tools information:
 Tools package installed: unknown
 File locking method: fcntl(2) (probably network-safe)
Mode |  UID  |  GID  |   Size  |    Modified Date   | Name
0755 | 93824 | 4871  |  199640 | 27-Jul-2021 21:41:24 | specmake
0755 | 93824 | 4871  | 1395560 | 27-Jul-2021 21:41:24 | specperl
0755 | 93824 | 4871  |   31240 | 27-Jul-2021 21:41:29 | specinvoke
0755 | 93824 | 4871  |   31240 | 27-Jul-2021 21:41:29 | specinvoke_pm
0755 | 93824 | 4871  |  228088 | 27-Jul-2021 21:41:24 | specxz
0755 | 93824 | 4871  |   39496 | 27-Jul-2021 21:41:29 | specmd5sum
0755 | 93824 | 4871  |    7747 | 27-Jul-2021 21:41:39 | specdiff
0755 | 93824 | 4871  |   89849 | 27-Jul-2021 21:41:39 | specpp
0751 | 93824 | 4871  |  263968 | 27-Jul-2021 21:41:24 | specrxp
0755 | 93824 | 4871  |  129163 | 27-Jul-2021 21:41:39 | runspec

The runspec --test command will run the perl test suite -- if all tests are successful, at the end you should see a message similar to the following:

All tests successful.
u=0.73  s=0.40  cu=10.50  cs=3.92  scripts=217  tests=71688

If we don't get any errors at this point, we can go ahead to the next step: building and running the HS06 benchmarks.

2. Building and running the HS06 benchmarks

If everything went well during the installation step, we are now ready to build and run the benchmarking executables.

SPEC CPU2006 provides a number of benchmarking routines, as well as several bsets (a bset is a named collection that contains several benchmarks). You can find all the available benchmarks in the directory $SPECDIR/benchspec/CPU2006/:

400.perlbench  435.gromacs    454.calculix    471.omnetpp    CINT2006.bset
401.bzip2      436.cactusADM  456.hmmer       473.astar      all_c.bset
403.gcc        437.leslie3d   458.sjeng       481.wrf        all_cpp.bset
410.bwaves     444.namd       459.GemsFDTD    482.sphinx3    all_fortran.bset
416.gamess     445.gobmk      462.libquantum  483.xalancbmk  all_mixed.bset
429.mcf        447.dealII     464.h264ref     998.specrand
433.milc       450.soplex     465.tonto       999.specrand
434.zeusmp     453.povray     470.lbm         CFP2006.bset

Optional: If interested, you can find detailed descriptions for all the benchmarks provided by SPEC CPU2006 at the following link.

HEP-SPEC06 consists of the all_cpp bset from SPEC CPU2006 together with the HEP specific configuration files *_cern.cfg (which specify the compilers, optimizations, flags to be used when building/running the benchmarks). In particular, the all_cpp bset contains all the C++ benchmarks that SPEC CPU2006 provides: 444.namd, 447.dealII, 450.soplex, 453.povray (these four are floating point benchmarks) and 471.omnetpp, 473.astar, 483.xalancbmk (these three are integer benchmarks).

Building the benchmark binaries

First, we have to copy the configuration files that we extracted from the HEPiX archive to the config folder of $SPECDIR:

cd speccpu2006-1.2
cp ../linux32-gcc_cern.cfg config/
cp ../linux64-gcc_cern.cfg config/

Now we can build the benchmarking executables with the following commands:

cd speccpu2006-1.2 # make sure we are inside $SPECDIR
# source the script to get the correct env vars, for e.g. to find runspec
. ./shrc
which runspec # should give $SPECDIR/bin/runspec

# make the executables
runspec --config=linux64-gcc_cern --action=scrub all_cpp
runspec --config=linux64-gcc_cern --action=build all_cpp

The example above shows how to build the 64-bit benchmark binaries -- if you want to build the 32-bit benchmark binaries you would use the linux32-gcc_cern.cfg file instead.

This build process will also take quite a bit of time to finish. At the end, you should see a message that indicates a successful build:

Build successes: 444.namd(base), 447.dealII(base), 450.soplex(base), 453.povray(base),
471.omnetpp(base), 473.astar(base), 483.xalancbmk(base)

Build Complete

If there are any errors during the build, the message will indicate which benchmark(s) failed and you can then look through the produced log files (which will be written to $SPECDIR/result/) to figure out what went wrong.

Running the all_cpp benchmarks

Now we can finally run the suite to obtain the HS06 score. Make sure you are the only user that is using the system you want to test, otherwise the benchmark results won't be valid.

For the following, we are interested in computing the HS06 score per physical CPU/core of the test system.

In principle, you want to run as many parallel copies of the benchmarks as the number of CPUs/cores on your system (i.e. run one copy of the benchmark per CPU/core). You can do this by using the following script (which you can place inside of $SPECDIR):

#!/bin/bash

echo 'job start'
START=`date`
echo "start time: " $START
COUNT=32 # change this to however many parallel benchmarks you want to run
echo $COUNT "copies of the benchmark will be started"
echo 'setup'
. ./shrc
echo 'runspec path is: '
which runspec # should give $SPECDIR/bin/runspec
echo 'run'
for i in $(seq $COUNT);
do
    runspec --config=linux64-gcc_cern --nobuild --noreportable all_cpp &
done
wait

END=`date`
echo "end time: " $END
echo "done"

Remember to use the same version of the config files that was used to build the benchmarking executables! Also note that the script starts 32 parallel copies of the benchmarks since there are 32 cores on the test system of these instructions.

The benchmarking process will take quite some time to run -- on the order of multiple hours, depending on the CPU under test. Once it has finished running, you should find several results files (CFP2006.NNN.ref.txt and CINT2006.NNN.ref.txt) in the $SPECDIR/result/ folder. You should have one set of these files for each benchmark copy that was run -- in the example above, this means there are 32 sets of these result files. The CFP2006.NNN.ref.txt files contain the results of the floating point benchmarks, while the CINT2006.NNN.ref.txt files contain the results of the integer benchmarks.

Computing the HS06 score

The runspec command, as given above, will run three iterations for each of the seven all_cpp benchmarks. The median of the three runs is used in the computation of the HS06 score.

In the results file, the medians are going to be marked with an asterisk, as shown below (excerpt taken from CFP2006.NNN.ref.txt):

444.namd         8020        314       25.6 *
444.namd         8020        314       25.6 S
444.namd         8020        314       25.5 S
447.dealII      11440        256       44.8 S
447.dealII      11440        260       44.0 *
447.dealII      11440        261       43.9 S
450.soplex       8340        176       47.4 S
450.soplex       8340        178       46.8 *
450.soplex       8340        179       46.6 S
453.povray       5320        122       43.6 S
453.povray       5320        121       44.1 S
453.povray       5320        121       43.8 *

The first column gives the name of the benchmark that was run. The second column gives the time (in seconds) that it takes to run the benchmark on the SPEC CPU2006 reference system. The third column gives the time (in seconds) that it took to run the benchmark on the test system. The fourth column gives the score (= second column / third column) for that particular iteration of the benchmark.

To compute the HS06 score for each core of the test system, we want to compute the geometric mean of the seven benchmarks. The total HS06 score for the test system is then given by summing up the HS06 score per core for each parallel run. To compute the average HS06 score per core, divide the total HS06 score by the number of cores/parallel runs.

The operations described above can be achieved by using the following script compute_score.sh, which can be run as ./compute_score.sh $SPECDIR/result/ . The credit for this script goes to the people who wrote the helper scripts inside the spec2k6-2.24.tar.gz archive from HEPiX (Alex Iribarren and Manfred Alef, as far as I could tell), I just took bits and pieces and put them together in this simpler script:

#!/bin/bash

# compute_score.sh

# Get a list of runs to look for
RUNS=""
NRUNS=0
for n in $1/CFP2006.*.ref.txt;
do
	RUNS="$RUNS `echo $n | sed 's/^.*CFP2006\.\(\w\+\)\.ref.txt/\1/'`"
	NRUNS=$(($NRUNS + 1))
done

# Now look through all the files and gather the partial scores
SUM=0
for n in $RUNS;
do
	partial=0
	count=0
	# awful looking sed expression to pick out the median score per benchmark
	for b in `sed -n -e '/^=\+$/,/^ Est. SPEC/!d; s/[0-9]\{3\}\.\w\+\s\+[0-9]\+\s\+[0-9]\+\s\+\([0-9.]\+\)\s\+\*/\1/p' $1/*.$n.*txt 2>/dev/null`;
	do
		partial="$partial + l($b)"
		count=$(($count + 1))
	done
	if [[ $partial != 0 ]];
	then
		SUM="$SUM + `echo "scale=8; e(($partial) / $count)" | bc -l`"
	fi
done

# Add up all the geometric averages / core
echo "Looked for results in directory $1"
SUM=`echo "scale=3; ($SUM)/1" | bc`
echo "Final score: $SUM for $NRUNS cores"
AVGSCORE=`echo "scale=3; ($SUM)/($NRUNS)" | bc`
echo "Average score: $AVGSCORE"

This is an example of the output produced by running the compute_score.sh script on the example system for this tutorial:

Looked for results in directory result_32_nid00786
Final score: 621.940 for 32 cores
Average score: 19.435

So we would say that the test system has an average HS06 score of 19.435 per physical core.

A short note on hyperthreading

Most modern CPUs will have hyperthreading turned on -- i.e. they will have 2*N threads (logical CPUs/cores) available.

If you find instructions for how to run HS06 on other websites, you might find that the recommendation is to run as many parallel benchmark runs as there are logical cores available on the test system.

The decision of how many parallel benchmark copies to run will depend on your specific use case and which aspect of your system you want to benchmark.

For example, if you want to get the HS06 performance of the machine/node as a whole (where the expectation is that every thread on the machine will eventually be filled up with jobs, as is the goal of a batch environment), then you probably want to use the number of logical cores available.

If, as shown in this tutorial, you are interested in the HS06 score per physical core (e.g. because you want to compare the performance of a single-threaded program between two different machines), then you probably want to use the number of physical cores available.

Another thing to note is that hyperthreading doesn't seem to result in a 2X improvement of the HS06 score -- that is, when tested on the example system used for this tutorial, the total HS06 score when running with all logical cores was found to be on the order of 20% higher than the total HS06 score when running with all physical cores.

-- IrinaEne - 2021-09-01