Week of 100322

Daily WLCG Operations Call details

To join the call, at 15.00 CE(S)T Monday to Friday inclusive (in CERN 513 R-068) do one of the following:

  1. Dial +41227676000 (Main) and enter access code 0119168, or
  2. To have the system call you, click here
  3. The scod rota for the next few weeks is at ScodRota

WLCG Service Incidents, Interventions and Availability, Change / Risk Assessments

VO Summaries of Site Availability SIRs & Broadcasts Change assessments
ALICE ATLAS CMS LHCb WLCG Service Incident Reports Broadcast archive CASTOR Change Assessments

General Information

General Information GGUS Information LHC Machine Information
CERN IT status board M/W PPSCoordinationWorkLog WLCG Baseline Versions WLCG Blogs   GgusInformation Sharepoint site - Cooldown Status - News


Attendance: local(Jaroslava, Harry(chair), Patricia, Andrea, Eva, Lola, Jean-Philippe, Malik, DavidG, MariaDZ);remote(Jon(FNAL), Michael(BNL), Jos(KIT), Gang(ASGC), Rolf(IN2P3), Thomas(NDGF), Onno(NL-T1), Rob(OSG), Tiziana(INFN), Gareth(RAL), Jeremy(GridPP)).

Experiments round table:

Weekend issues: IT/INFN-T1 (Fri Mar 19), GGUS#56583: Problems with StoRM connection, two storm front-end were down, jobs were failing. FEs restart helped.

NL/NIKHEF-ELPROD (Fri Mar 19), GGUS#56587: Two of the disk servers behind tbn18.nikhef.nl have problems accessing their filesystems. Fix requires vendor support, they have been notified.

DE/WUPPERTALPROD (Sat Mar 20), GGUS#56590: Source file/user checksum mismatch observed. Corrupted network link between CC and uni backbone, recognised on Friday and switched to backup link. Since then link is stable, no coruptions. Consistency check of all files in progress. This issue is internal T2 issue, it has nothing to do with the LHC OPN.

NL/SARA (Mon Mar 22): FTS migration to v.2.2.3 with downtime till 13.00.

T0 Highlights: All T0-->T1 transfers migrated to use FTS2.2.3 fts22-t0-export.cern.ch

T1 Highlights: MC processing completed at RAL, running at PIC

Backfill test jobs running giving a low levels of failures at IN2P3 (out of memory), CNAF (storage issue, see below) and ASGC (under investigation).

CNAF: On Friday afternoon, StoRM issues affecting transfers and backfill test jobs, fixed around 8 pm. Tiziana reported that this was due to a diskserver with hardware issues that will be corrected very shortly.

KIT: LoadTest transfers to T2_US_Wisconsin (GLOW) failing with SRM authentication issues, under investigation.

T2 highlights: MC ongoing in all T2 regions.

Job failures in UK and US T2s, resubmissions successful.

196 MC files lost at T2_US_Nebraska (nebraska), invalidated.

GENERAL INFORMATION: Production ongoing at all sites with two new MC cycle also running. Starting this week will make contact with the 15 ALICE sites which are not providing a CREAM-CE. The goal is to have the system ready and deployed at all sites for the 31st of May. Tiziana queried the status for Italy - the reply was that only Cagliari is missing and they are being followed up. There is also some flexibility in this date.

T0 site: Pass1 reconstruction operations still ongoing. Setup defined through 3 VOBOXES defined in redundant mode, both the LCG-CE and the CREAM-CE backends nicely performing.

T1 sites: Pass5 and 6 operations ongoing at the Tier1 sites. No incidents to report during the Weekend

T2 sites: GRIF_IRFU: The information provider of the local CREAM-CE still in bad state. The lastest news confirmed the system should be back in production today

Prague-T2: The site admin announced error messages in Monalisa.log file. Following the issue with the experts.

Experiment activities: LHCb Tier 1 Jamboree in Amsterdam - this afternoon + tomorrow morning.

T1 sites issues: NIKHEF all jobs fail systematically to upload data to everywhere. Sound like jobs use GID =0 that is associated to root. GGUS ticket open.

Sites / Services round table:

  • KIT: Had a strange network access problem Saturday affecting connectivity from outside e.g. GGUS was unreachable. However the OPN was not affected nor apparently any running services. The problem cleared at about 17.00 Saturday.

  • NL-T1: 1) NIKHEF is still having disk server problems - no news yet. 2) The LHCb issue of failing jobs (GGUS 56581) is under investigation. 3) FTS at Sara is now running on new hardware and under version 2.2.3. Working correctly apart from the monitoring agent which has problems with its php scripts which should be fixed by tomorrow. Apologies for putting the wrong downtime in the GOCDB.

  • CERN CASTOR: Following the successful Hammercloud testing of the new xroot plugin for Castor, it is proposed to upgrade the main Castor Atlas redirector on Wednesday 24th at 10:00. This upgrade will apply the same code version which has been run for two weeks with Panda to the main castoratlas server.

Change assessment is at https://twiki.cern.ch/twiki/bin/view/CASTORService/ChangesCASTORATLASXroot2193For102.

The timetable proposed is Wednesday 24th at 10:00 with around 1 second of downtime during the upgrade. If there are any concerns, please raise them (to Tim.Bell@cernNOSPAMPLEASE.ch) before midday on Tuesday 23rd.

  • CERN firewall: There will be a maintenance on Wednesday between 07.30 and 13.30 with two short interruptions but the OPN and HTAR routes used by grid services will not be affected.

AOB: MariaDZ noted there is only one USAG meeting left under the auspices of EGEE3. This will hence be an important meeting to clarify how to proceed with this level of user support post-EGEE. She proposed to hold this during the EGEE user forum in Upsalla, assuming access to a conference phone facility, where there should be a good attendance already. Her proposal is 16.00 CEST on 14 April and would like any objections to be emailed to her (Maria.Dimou@cernNOSPAMPLEASE.ch) as soon as possible.


Attendance: local(Harry(chair), Jaroslava, Miguel, Nicolo, Eva, Jean-Philippe, Alessandro, Lola, MariaDZ);remote(Jon(FNAL), Tiziana(INFN), Michael(BNL), Josep(CMS), John(RAL), Ronald(NL-T1), Rob(OSG), Gang(ASGC)).

Experiments round table:

DE/FZK-LCG2 (GGUS#56618): Failing transfers to T2s with "LOCALITY UNAVAILABLE", 2 ATLAS disk-only hosts are down. In progress.

IT/INFN-T1 (GGUS#56625): Failing transfers to INFN-T1. Errors in StoRM connections were due to problems on atlas gpfs cluster, because of failing interconnections. Solved.

CA/TRIUMF: Wrong dir naming convention for groupmc09 and tape area, cause under investigation.

NL/SARA: FTS successfully upgraded to 2.2.3.

From an EGEE broadcast: The ATLAS VO ID Card has been updated with the new requirements for the Batch Systems. All the sites are kindly requested to check the new requirements, and in particular for what concerns:

- Scratch space/slot: 20GB;

- CPU Time: 48 hours / 8 HS06;

- Wallclock Time: 52 hours / 8 HS06;

- VMEM: 3GB, but preferably no limit.

For any question please send an email to atlas-project-adc-operations@cernNOSPAMPLEASE.ch

MC processing completing at PIC

Backfill test jobs running

PIC: Consistency check failure for custodial data (3600 files) hosted at PIC - ticket opened on site contacts to verify PhEDEx consistency agent logs. Detailed report on progresses on tickets:

T2 highlights: MC ongoing in all T2 regions.

Test workflow submitted to CNAF preproduction WMS + Pisa CREAM CEs to verify fix of WMS-CREAM job reporting issues.

SAM test failures at T2_RU_ITEP (ITEP): old CMSSW version needed for SAM tests not installed anymore; will be solved by migrating SAM tests to a recent CMSSW version.

  • CMS Weekly-scope Operations plan

Data Ops

Tier-0: prepare for collisions. Tier-1: backfill and waiting for pre-production requests. Tier-2: finishing up requests.

Facilities Ops

SAM and JobRobot Datasets subscribed to T1s and T2s took only 1 day to replicate in almost all the sites. Approximately 60% of the transfers were directly from T1_CH_CERN during the first 16 hours. To inspect central PhEDEx FilePump logs from last week and run the transfer latency tools on them to find out where the rest of the files came from.

CERN FTS 2.2.3 endpoint upgrade in all T1 agents, in all PhEDEx instances. Running ok.

Reminder to sites to emove -myproxy/-passfile options from PhEDEx Download agent. Some sites asked to upgrade to latest PhEDEx version, this week.

We plan to move from Savannah to GGUS Team tickets for T1s, always opened by the CRC, in case of problems identified by the CSP's at the T1s. Instructions for CRCs/CSPs will be adapted shortly.

WLCG and our current CMS tools to trace Downtimes only take into account SCHEDULED DOWNTIMES OUTAGEs. AT_RISK SD's are not accounted as maintenance periods. The same WLCG rules apply for CMS SD's: http://goc.grid.sinica.edu.tw/gocwiki/GOCDB_User_Documentation.

  • ALICE reports - GENERAL INFORMATION: Two MC cycles running simultaneously to raw reconstruction (Pass1 at the T0) and data reprocessing (Pass5 and 6 at T1 sites).

T0 site: Exclusively performing pass1 reconstruction activities. No fundamental issues to report

T1 sites: ALICE T1 sites all in production

T2 sites: PNPI (Russia): CREAM-CE provided yesterday, it has entered production this morning. The site has been configured in CREAM-submission mode only

GRIF_IRFU (France): Still we do not have news about the problem reported last week cocerning the resource info provider (the system is stucked and the site has been put back in WMS submission mode)

SpbSu (Russia): The 2nd VOBOX has been put in maintenance mode (out of ALICE production) to perform several tests with the system

Sites / Services round table:

  • KIT (by email): Access problem for ATLAS T0D1 data because of broken disk controller. Expect to have this fixed at 16:00. Failover to redundant path failed for unknown reason which is being investigated.

  • NL-T1: Last issues after FTS upgrade resolved but the monitoring agent is still being worked on (CNAF offered to share their experiences on this). All disk servers have been restored with no loss of data but are still currently in read-only mode. (By email: there is however 180 TB of older writeable disk).

  • INFN: The ATLAS gpfs problem was one of interconnection that was fixed by restarting processes but is not yet fully understood. Only ATLAS was affected.

  • OSG: Will be upgrading their testbed ggus ticketing interface bringing a cleanup of the site names and leading to a future production system upgrade.

AOB: CERN is organising a first physics event with media invited to be present on the day for the first 7 TeV collisions. The date has been confirmed to be Tuesday the 30th March.


Attendance: local(Miguel, Jaroslava, Harry(chair), Nicolo, Eduardo, Steve, Eva, Nilo, Jean-Philippe, Alessandro, Lola, MariaDZ);remote(Jon(FNAL), Tiziana(CNAF), Michael(BNL), Gang(ASGC), Onno(NL-T1), Angela(KIT), Thomas(NDGF), Gareth(RAL), Rolf(IN2P3), Rob(OSG)).

Experiments round table:

Fast track muon reprocessing is finished, INFN and BNL performed perfectly! Reprocessing was done in 36 hrs. Thanks!

DE/FZK fixed their disk-only hosts (GGUS#56618). Angela reported there was a controller problem where removing a controller from what is normally a redundant setup led to other parts misbehaving.

ASGC migrated FTS to 2.2.3 successfully in the morning, TRIUMF will migrate FTS in the evening. Suggest that sites having trouble setting up the monitoring agent tell other clouds (and also wlcg-scod@cernNOSPAMPLEASE.ch).

T0 Highlights: Reduced CASTORCMS_DEFAULT SLS availability - related to large number of queued user requests Queued transfers

GGUS #56666 - srm-cms.cern.ch no longer listed as SRMv2 in SAMDB, so not receiving CMS SAM tests. Seems related to change in BDII publication of the service.

T1 Highlights: MC processing completing at PIC, 5000 more jobs to run

Backfill test jobs running

CNAF: SRM endpoint restarted

T2 highlights: MC ongoing in all T2 regions.

New generator-level input files produced at CERN

SAM test failures at T2_RU_ITEP (ITEP): old CMSSW version needed for JobRobot tests reinstalled

T2_PK_NCP (NCP-LCG2) reporting power issues, services down.

T2_US_Florida (uflorida-pg, uflorida-ihepa, uflorida-hpc), T2_US_Nebraska (nebraska) , T2_US_Purdue (purdue-steele, purdue-rcac) - MC production files lost or corrupted, being invalidated.

All: CE-cms-analysis test migrated to use new CMSSW version for SLC5 - test will return WARNING status instead of ERROR during migration period to let sites fix issues. Tiziana queried what is new - reply was a new input dataset and software version. A few sites - no Tier 1 - have things to fix and will be followed up.

  • ALICE reports - GENERAL INFORMATION: reconstruction and MC production activities going on at all sites. In addition, the testing of the new AliEnv2.18 has started with 3 candidates: Subatech, RRC-KI (these two already available) and ITEP (installation problems). Massive distribution of the new version expected for the next week. MariaDZ suggested ALICE use ggus tickets to track such installations as there is good workflow tracking.

T0 site: Both submission backends working with no particular issues to report

T1 sites: RAL production stuck this morning. A restart of all ALICE services at the VOBOX has been performed. Rest of T1 sites are in production

T2 sites: AliEnv2.18 now available at Subatech (France), RRC-KI (Russia). in addition Subatech is back in production after the AliEn update with no incidents to report

Next candidate to test AliEnv2.18 is ITEP (Russia). The site admin has been informed this morning that the sw area is not accesible. Following the issue with the local experts

Testing the new sw infrastructure in Athens. The site has finished the migration to SL5 and will be put back in production by this week

Experiment activities: No production activities. Main activity devoted to setup a suite to evaluate xrootd for the long standing file access problem that LHCB is facing at sites. Direct CREAMCE submission: a first prototype has been setup at CNAF. This is probably the way LHCb will eventually submit batch work.

T1 sites issues: NIKHEF all jobs fail systematically to upload data to everywhere. Sound like jobs use GID =0 that is associated to root. May be a worker node configuration problem. GGUS ticket open. Onno reported that the ticket 56581 is waiting for a reply from LHCb for extra information - Roberto will follow-up.

CNAF: CREAMCE problems submitting with Role=pilot due to too few pool accounts setup there. Tiziana will follow-up.

* T2 sites issues: GRISU-CYBERSAR-CAGLIARI pilot aborting - ticket submitted.

Sites / Services round table:

  • ASGC: File transfers working fine after the FTS 2.2.3 migration.

  • BNL: Still seeing the issue of many concurrent ATLAS jobs opening/writing files exhausting connections to the LFC. A production independent testbed has now been setup which demonstrates the production behaviour lies with clients holding open connections. Currently the maximum number of threads (hence clients) is compiled to be 99 so BNL plan to recompile this to around 500. J-P.Baud suggested to also look at using Oracle connection pools on the backend DB server as otherwise each front-end client would require a separate connection and these are memory expensive. This may, however, have an impact back on the client side. [Note, it was confirmed by the DBA after the meeting that connection pooling is already in use for this application]. These solutions address two separate parts of this problem and J-P and Michael will stay in touch on this issue.

  • NDGF: Some dcache servers dropped out of the bdii publishing causing SAM tests to fail - not clear that users were affected. Scheduled dcache upgrades today took a little longer than planned.

  • RAL: Have a scheduled LFC outage tomorrow but not for an LHC VO.

  • OSG: Two tickets just arrived on problems transferring files from Austria to Nebraska and Florida. Being looked at.

  • CERN FTS: The scheduled FTS-T2-SERVICE upgrade to FTS 2.2.3 took 4 hours instead of 2. It was noticed that the monitoring agent installation has been broken for some time.

  • CERN networking: Scheduled tests of a new external firewall have gone very well so this will be left in place for a few days longer than was previously announced.

AOB: (MariaDZ) The date of the next (and last) USAG meeting must change (no tel. facility free during the User Forum). April 21st 16hrs CEST (CERN room and usual audioconf) is the new date/time. Here is the agenda, this is the last meeting: http://indico.cern.ch/conferenceDisplay.py?confId=88353 Please let maria.dimou@cernNOSPAMPLEASE.ch know now if there is a major problem with this date.

LFC LHCB@CNAF: Jean-Philippe has looked at a GGUS problem ticket 56650 CNAF submitted after an upgrade of the LFC server for LHCb. The error message seen does not come from the LFC nor Oracle streams neither is it a Yaim problem. He suspects some local procedure and is requesting more information from CNAF. Tiziana will follow up.


Attendance: local(Jaroslava, Harry(chair), Eva, Andrea, Malik, Jean-Philippe, Gavin, JanI, Julia, Patricia, Roberto, Lola, Alessandro, MariaDZ, Jamie, MariaG, Steve);remote(Jon(FNAL), Gonzalo(PIC), Alessandro(CNAF), Gang(ASGC), Gareth(RAL), Rolf(IN2P3), Rob(OSG), Ronald(NL-T1), Thomas(NDGF), Jeremy(GridPP), Angela(KIT)).

Experiments round table:

PIC cloud will have a long planned scheduled downtime during the LHC Media Day (30th March)

We have activated FTS checksums for Taiwan and Canadian clouds.

T0 Highlights: Preparing for collisions and replay tests; Prompt reconstruction. Job robot (submitting monitoring test jobs) was found to be stuck with a hanging process since yesterday.

T1 Highlights: Backfill jobs running at all sites apart from PIC (which is running real production) and release validation jobs running at FNAL

T2 highlights: MC production ongoing

T2_BR_UERJ was auto-deleting all files after transfers (this is meant for debugging mode); fixed in the PhEDEx configuration

Transfers fro T2_BR_SPRACE failed with authentication errors; it was due to a reconfiguration of the SE

Transfers from PIC to Caltech are failing; under investigation

A routing problem caused the failure of the squid and Frontier SAM tests in T2_AT_Vienna

GENERAL INFORMATION: latests MC cycle production has finished. Jut some few remaining jobs are running currently on the system.

Following the CREAM setup requirement to be ready at all sites before the 31st of May, at this moment, only 9 sites out of the full production are not providing the mentioned service. 5 sites have guarranteed to setup the system by the end of April.

T0 site: ce201.cern.ch (CREAM-CE) is not performing ok (timeout messages at submission time). Although ALICE keeps on using the ce202.cern.ch for submission purposes, 2 VOBOXEs out of 3 at the Tier-0 remain with a single CREAM-CE submission backend. GGUS ticket submitted: 56710

T1 sites: NIKHEF: The services have been rechecked this morning taking advantage of the low number of jobs currently running. The site admin has provided us with the list of all issues found in CREM which prevent the setup of the system in production. Following the issue with the site admins and experts at the site.

SARA: CREAM-CE at the site showing submission problems. GGUS ticket submitted (#56709)

T2 sites: Poznan (Poland): Site has been migrated to CREAM backend this morning. site back in production

UNAM (Mexico): Same approach as for Poznan

Wuhan (China): Checking the whole site setup before data taking by this afternoon. The aim is that the site will be in production in time

  • LHCb reports - No production activity just the usual few thousand user jobs run per day.

T1 sites issues: CNAF- Storm: LCMAPS problem causing some users to fail their jobs in uploading data and SAM test jobs too. This is a known bug on LCMAPS (when log files bigger than 2GB).

CNAF: CREAMCE : problems submitting with Role=pilot due to too few pool account setup there. fixed by increasing to 50 the pool accounts for pilot role.

NIKHEF: problem with Group ID passed to LFC clients being zero (root). Under investigation by DIRAC experts.

PIC: Still down for LHCb batch with a software storage module problem on worker nodes - a problem of DIRAC and the operating system flavour.

Sites / Services round table:

  • PIC: Reminder of long planned scheduled downtime next Tuesday - unfortunate clash with CERN media day.

  • FNAL: Unclear as to whether WLCG will run an OPS SAM test against a site that is publishing that it has glexec capability. We know how how to publish and locally monitor this capability, it is just a question of what OPS SAM tests are available for this. Steve Traylen reported some information on this was given by Maarten Litmaath at yesterdays GDB meeting. We will try to get a clear answer and report back tomorrow.

  • CNAF: The problem reported yesterday of a strange LFC error message of non-compilation has been investigated and found to be some old unimportant procedure that will be cleaned up.

  • NL-T1: There was a scheduled network maintenance at SARA from 10 to 10.30 with a brief LHCOPN interruption on the direct CERN link but the backup link via NDGF worked as it should.

AOB: LHC is preparing for the planned first collisions at 7 Tev next Tuesday.


Attendance: local(Harry(chair), Jaroslava, Alessandro, Jan, Eva, Jean-Philippe, Malik, Roberto, Patricia, Lola, Gavin, Julia, Miguel);remote(Jon(FNAL), Rob(OSG), Tiziana(CNAF),Onno(NL-T1), Jos(KIT), Jeremey(GridPP), Rolf(IN2P3), Gareth(RAL), Pepe(CMS)).

Experiments round table:

  • ATLAS reports - New Central catalogs put into production yesterday.

T0 Highlights: Preparing for collisions and replay tests. Prompt reconstruction.

T0_CH_CERN - GGUS #56666 has been closed - srm-cms.cern.ch no longer listed as SRMv2 in SAMDB, so not receiving CMS SAM tests. There was a confusion with GlueServiceAccessControlBaseRule (still there) and GlueServiceAccessControlRule (deprecated). The former was added to the info provider, and the endpoint shows up in the BDII.

T1 Highlights: Backfill jobs running at all sites but PIC (which is running real production). Release validation at FNAL.

T2 highlights: MC production ongoing.

T2_BR_UERJ was auto-deleting all files after transfers; fixed in the PhEDEx configuration. Some lost datasets have been subscribed again, and we plan to run a full consistency check when the re-transfers end.

Transfers from PIC to Caltech are failing; PIC admins notified.

A routing problem caused the failure of the squid and Frontier SAM tests in T2_AT_Vienna.

DDT-team started a round of T2-T2 data transfer links commissioning.

All: The Job Robot was found not to be submitting jobs since 24th/March; it was due to a hanging process. Normal job flow resumed after manually killing the process. Today it is confirmed the JobRobot is running happily at all the sites.

GENERAL INFORMATION: Low number of jobs currently in production. The latest MC cycle is completed, therefore some uncheduled user analysis and Pass1 reconstruction jobs are the current production activities.

T0 site: No issues observed. Both submission backends currently working.

T1 sites: SARA ticket submitted yesterday concerning the local CREAM-CE system (authorization problems at submission time) has been solved and verified this morning. The system is back in production. The rest of T1 sites are all in production.

T2 sites: Wuhan - Wrong information provided by the local CREAM-CE, announced yesterday during the ALICE TF Meeting. Instabilities expected today at ITEP while configuring the 2 available VOBOXES in failover mode with a view to rolling this out to other ALICE sites next week.

  • LHCb reports - No production activity just the usual few thousand user jobs run per day. xrootd first tests at CERN.

T0 sites issues: It looks like reading a file with xrootd at CERN is OK from lxplus but does not seem to work from an lxbatch node (file open but not possible to read). Can it be there a firewall issue blocking the data channel between the WN and the xrootd server ? Miguel rather suspected authorisation issues.

T1 sites issues: CNAF can go ahead with the intervention on the ConditionDB on Monday. Suggestion to prepare for the intervention to minimize the downtime period

T2 sites issues: CSCS-LCG2 pilots stalled.

Sites / Services round table:

KIT: Two ATLAS dcache pools are down with a controller problem (different from the one earlier this week). Will be fixed by the end of today.

RAL: There will be a short 'at-risk' period on Monday while LSF license keys are changed.

OSG: Have discovered that there is a limit of 4K characters allowed in the ticket information field when entering OSG tickets into GGUS (truncation occurs). Can easily be passed with log outputs etc. Will follow up with Maria.

CNAF: Alessandro di Girolomo queried the status of the upgrade of the ATLAS conditions DB at CNAF (Note that of LHCb is being done next Monday). Eva reported that this was waiting on ATLAS to specify a date. Her contact at CNAF was Alessandro Cavalli - not there today. Alessandro will get in touch with him Monday to understand the impact and plan a date. Eva reminded that this is a non-transparent intervention.


  • GLEXEC deployment: Following queries yesterday here are Maarten Litmaath's detailed instructions:

1. Install and configure glexec for use by the supported LHC experiments

- the supported VOMS roles are /atlas/Role=pilot etc. - the CE needs to map those roles to dedicated (pool) accounts - the WN must have a glexec.conf with those accounts whitelisted - for EGEE/gLite the necessary configuration steps on the WN are detailed as part of the GLEXEC_wn documentation, e.g. here:


2. For EGEE/gLite: get the Nagios tests to succeed. Link:


For OSG: get the equivalent tests to succeed, but they first need to be made available in their test infrastructure. I will follow up.

3. When the tests succeed or when glexec is believed to be configured correctly for the supported experiments, let the site's CE(s) publish an extra capability:

GlueCECapability: glexec

  • Summer time starts in Europe at 02:00 (local time!) Sunday morning.
    Key: blue=DST observved, orange=no longer observed, red=never observed. DST.png

-- JamieShiers - 22-Mar-2010

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng DST.png r1 manage 46.8 K 2010-03-22 - 10:20 JamieShiers  
Edit | Attach | Watch | Print version | History: r17 < r16 < r15 < r14 < r13 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r17 - 2010-03-26 - HarryRenshall
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback