Achieved Rates - Daily Averages for Disk-Disk Transfers

• Rates achieved as per Gridview - see SC4blog for more information on events, interventions, changes etc.

• See also SC4 Wan traffic and disk - tape status page

• Note that the official goal is to meet or exceed the target rate every day (or make up for it shortly after - more information below). In addition, the startup (ramp-up to full nominal rates) needs to be rapid - essentially a step function at the beginning of each LHC running period.

• BOLD - sites meeting or exceeding their nominal target for that day

• ITALICS - sites within 15% of their nominal target for that day

Site-specific Issues and Plans

CNAF

• As far as I understand, the CASTOR2 LSF plugin and rmmaster version you are running is leaking jobs. We had the same problem at CERN during the SC3 re-run in January. The symptom is that LSF dispatches some fraction of the jobs without the message box 4, which is posted from the plugin. It is briefly described on slide 4 in the CASTOR2 training session 5: http://indico.cern.ch/getFile.py/access?resId=33&materialId=0&confId=a058153. Unfortunately it only describes a workaround for 'get' requests, while you are seeing the problem for 'put'. The procedure should be similar but you have to decide the filesystem yourself (for 'get' it is much simpler since the list of target filesystems is already posted in msg box 1). Alternatively you may just post an empty or non-existing filesystem, which will cause the job to fail. Simply killing the job with bkill would also work but result in an accumulation of rows with STATUS=6 in the SUBREQUESST table. The problem has been fixed (a workaround for a timewindow in LSF) in 2.0.3 so the best would be to upgrade. Olof

• We are upgrading CASTOR2 to the latest release and we should be ready to repeat the disk to disk transfer tests during the first half of May. luca

FZK

• The link to FZK should be moved to the dedicated circuit in the timeframe of early June I think. For the moment the IP service of GEANT has a limitation of single streams <= 1Gb/sec dure to packet re-ordering in the M160 routers. However, I am not sure if this would be a problem, in reality, in the multi-stream usage in SC4. However, traffic via GEANT-IP will be using a shared network so performance may vary. David

• We are in the progress of analyzing the limiting factors. Some key (network, dcache) people are not present this week. A report and throughput improvement planning will arrive next week (wednesday). First things that come to mind: There is a difference in the setup as compared the one used in SC3. Most notably the 2.6 kernel on the gridftp receivers. Another cause could be the costs computation dcache performs. New nodes received an inordinate amount of traffic and left the remaining nodes idling. This could be tuned better. I do not think the single streams limitation would be the real problem because as far as I know the link runs via the same path and hardware as with SC3. We did not see the problem then. Correct me if I'm wrong. Jos

• Due to Easter holidays and the ongoing work on our tape connection at GridKa we won't be able to start the tape tests before mid of next week. Doris

• ... so we will continue to monitor disk - disk transfers ... Jamie

IN2P3

• No theories yet - configuration at IN2P3 should be able to handle up to 300MB/s.

• No limitating factors have been found at IN2P3-CC during SC4 disk-disk transfers except a memory issue on the SRM node which lead to a drop in the rate every night at 21:40 GMT during the DB backup (20 min). Memory will be upgraded and IN2P3 will study the possibility to distribute dCache core services on 2 different hosts. Another possible issue is the concurrent access to the SRM by Atlas. This has to be investigated. It seems in most of the cases, FTS was not able to send enough data with the initial number of files in the channel (15) when there was a concurrence between all sites: with 15 files at the beginning of the challenge, IN2P3 could sustain > 200MB/s, whereas one week later, the rate was half of that with 30 files. When increasing to 45, the rate immediately doubled but then started to decrease again. Moreover we noticed that the number of concurrent transfers was often lower then the number of files set in the channel. Lionel

ASGC

• Target here is to reach stable transfers at 50MB/s. Having sustained this over several days, we would try to ramp up to the full nominal rate by end May Jamie

Week two (April 10 on)

• Week 2 average to sum of Tier1 sites is 1262 MB/s - 79% of the target.

Week one (April 3 on)

• Week one summary: there is an improvement with time in the above: the total average daily rate out of CERN increases but then plateaus around 1200MB/s; the number of sites participating builds up as does their stability. There is the clear indication that there is (are) a (some) limiting factor(s) at CERN which need to be understood before trying to push the rate up to the target.

Schedule & Targets for SC4

Tier0 - Tier1 Throughput tests

Disk - disk Tier0-Tier1 tests at the full nominal rate are scheduled for April.

The proposed schedule is as follows:

• April 3rd (Monday) - April 13th (Thursday before Easter) - sustain an average daily rate to each Tier1 at or above the full nominal rate. (This is the week of the GDB + HEPiX + LHC OPN meeting in Rome...)
• Any loss of average rate >= 10% needs to be:

1. accounted for (e.g. explanation / resolution in the operations log)
2. compensated for by a corresponding increase in rate in the following days

• We should continue to run at the same rates unattended over Easter weekend (14 - 16 April).
• From Tuesday April 18th - Monday April 24th we should perform the tape tests at the rates in the table below.
• From after the con-call on Monday April 24th until the end of the month experiment-driven transfers can be scheduled.

• To ensure a timely start on April 3rd, preparation with the sites needs to start now (March). dTeam transfers (low priority) will therefore commence to the same end-points as in the SC3 disk-disk re-run as soon as sites confirm their readiness.

• The nominal rate for PIC is 100MB/s, but will be limited by the WAN until ~November 2006.