During the combined LHCOPN/LHCONE meeting which took place in Washington on 13 & 14 June, 2011, questions were raised about why there were two, albeit supposedly compatible, versions of perfSONAR (perfSONAR-MDM and perfSONAR-PS). US-ATLAs gave very positive feedback on their experience with perfSONAR-PS, and Jason Zurawski (then of Internet2, now of ESnet) gave a presentation which included the US-ATLAS use-case (link).The distributed nature of the perfSONAR-ps solution was deemed attractive, as was the ease of installation (ISO-CD) and the work that US-ATLAS has done on visualisation (including alarm generation using Nagios). Following a strong request from ATLAS, it was decided that perfSONAR-ps would be installed in all the LHCOPN Tier0 and Tier1 sites. The Tier1s present were all in favour of the idea, and John Shade agreed to coordinate and drive the roll-out, with Jason Zurawski providing the technical expertise!

Installed Servers

Tier Type Hostname IP address Comments
RAL Latency:  
CC-IN2P3 Latency:  
CERN Latency:  
TRIUMF Latency:
SARA Latency:  
ASGC Latency:  
BNL Latency:  
CNAF Latency:  
NDGF Latency:  
PIC Latency:  
FNAL Latency:  
KIT Latency:  
KISTI-GSDC Latency:  


Tom Wlodek's orginal Experimental Independent perfSONAR Dashboard has been replaced by MaDDash, developed by ESnet. It includes a number of meshes in addition to the LHCOPN mesh - these can be accessed by clicking on "Dashboard" near the top of the page. More information on MadDash is available here.

As before, each site should ensure that its cells are coloured green!


After a fairly long period of manual installations and configurations, WLCG put considerable effort into developing automated packages for perfSONAR installations, and a number of other Twiki pages have sprung up. The most pertinent page is without doubt the WLCG perfSONAR Deployment Twiki created by Shawn McKee.

Up-to-date instructions for installing the latest version of perfSONAR and enabling mesh configurations for testing between WLCG sites can be found on this Twiki page.

Toolkit Page

The perfSONAR Performance Toolkit (pS-Performance Toolkit) is a customized version of a Knoppix Live-CD bootable disk. The software consists of an ISO disk image which provides a Linux distribution based on CentOS 5.5, and the perfSONAR software. The toolkit can be downloaded on disk or burned to CD - both versions are identical in terms of functionality, but netinstall has the added benefit of being able to 'yum update' new stuff as it is released.The main toolkit page is here:

The latest version of the toolkit is 3.3. Netinstall users can do a 'yum update' and then re-boot. LiveCD users will need to download a new ISO image from here.

The RPMs are located here.

There are instructions on the main page for enabling the yum repo (automated package management):

Instruction Manual

This contains EVERYTHING and may be a little dry, but is the perfSONAR-PS reference. A very useful FAQ is available here.

Firewall ports

In order for perfSONAR to be really useful for ad-hoc network troubleshooting, a number of firewall ports should be opened. The detailed list is given here.

Beware, however, that a yum update of the toolkit (perl-perfSONAR_PS-Toolkit-SystemEnvironment.noarch) will silently overwrite your node's firewall configuration in /etc/sysconfig/iptables! The toolkit developers have promised to try and improve matters in the future, but be sure to always read the release notes!


It is recommended to run throughput (BWCTL) and latency (OWAMP) tests on separate servers. This is because the throughput tests (iperf) use a lot of CPU, memory, and all of the available resources on a network card, thus negatively impacting the smaller packet streams of the latency tests. Choosing which component to install (Latency or Bandwidth) is a simple 'selectable' option as you configure the host. For servers running the bandwidth tests, we recommend a 10 Gbps NIC. Note that it is not recommended to run 10G bandwidth tests through a firewall.

NTP Servers

Which NTP server to use is site-specific. There are public NTP servers available, but it's better to synchronize with servers in close proximity, and that have the lowest possible stratum number (stratum0 = NTP server getting time from a GPS antenna; stratum1 = getting the signal from a stratum0, stratum2 = signal from stratum1, etc.).

Supported Platforms


Simple hardware works the best! For example, Internet2 uses 1U Dell servers that are about 5 years old for their testing. The recommendations are:

  • Compatible with 32 bit or 64 bit Intel x86 architecture
  • CPU Speed Requirement of 2.6 GHz for single core machines
  • CPU Speed Requirement of 1.8 GHz for multiple core machines
  • Main Memory Requirement of 2 Gigabytes
  • Storage Requirement of at least 250 Gigabytes
  • RAID level 0+1, or no RAID if possible
  • Network Card Speed Requirement of 1Gbps, as a "daughter" card (avoid on-motherboard NIC for performance testing)
  • Network cards to support multi-homing (as many as required)

Software and security

The perfSONAR-PS project currently only supports the CentOS 5 platform for x86 architecture. However, it may be possible to install and use these RPMs on other RHEL binary compatible systems like SL. If you feel like trying, please let us know how you get on!

For instructions on how to enable the yum repo:

In order to configure your servers such that only users in possession of a valid grid certificate can access the web pages, please see Virginie Longo's security guidelines (document attached to this page).


Although the importance of community strings is yet to be determined, we recommend configuring the following for starters: LHCOPN, LHCTier1, ATLAS, CMS, ALICE, LHCb, LHCONE.

Links to things of interest

Site Contacts

Tier Contact Person e-mail address
RAL Brian Davies
CC-IN2P3 Laurent Caillat-Vallet
CERN Stefan Stancu
TRIUMF Vitaliy Kondratenko
SARA Sander Boele
ASGC Wenshui Chen
BNL John Bigrow
CNAF Donato De Girolamo
NDGF Magnus Bergroth
PIC Fernando Lopez
FNAL Darryl Wohlt
KIT Aurélie Reymund
KISTI Sang-Un Ahn

CERN e-group mailing list: (includes the above names, plus Jason Zurawski and John Shade).

-- JohnShade - 20-Jun-2011

Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf 20110916-LHCONE_Walk.pdf r1 manage 888.6 K 2011-09-20 - 11:08 JohnShade Jason's per-site check, including some top-tips!
PDFpdf 20110920-LHCONE_Walk.pdf r1 manage 1260.9 K 2011-09-20 - 16:02 JohnShade Latest version of Jason's checks.
PDFpdf 20110921-LHCONE_Walk.pdf r1 manage 1265.6 K 2011-09-22 - 08:23 JohnShade Final(?) version
Texttxt perf-node-users_perfSONAR-PS_Toolkit_3.2.1_now_available.txt r1 manage 2.2 K 2011-10-19 - 09:43 JohnShade Announcement (with links) of perfSONAR-PS 3.2.1
PDFpdf perfSONAR-PS_Security_Guidelines.pdf r1 manage 625.4 K 2011-12-09 - 10:21 JohnShade perfSONAR-PS Security Guildelines for LHCOPN
Edit | Attach | Watch | Print version | History: r66 < r65 < r64 < r63 < r62 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r66 - 2018-12-19 - ShawnMcKee
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LHCOPN All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback