BUG:61586

BUG:55482

UPDATE AT THE MEETING: This will be fixed in gLite 3.2 but not in gLite 3.1. OCC will follow up.
08/02/2010There is now a fix for gLite 3.1, the bug is set to "Fix Certified", I think this action can be closed.

This is now done, started on Monday 15th of February.

*Update Feb 18th* More hardware was thrown at the problem and the situation is
resolved.

*Update Jan 31th* : Will be discussed by the ROC managers in Lyon next week (Tuesday 5th)

*Update Feb 18th*: Will add a link to minutes of ROC managers meeting.

*Update Feb 25th*:
This is the link: https://edms.cern.ch/file/893655/1/ROC-mgrs-05-02-2008(ARM-11).htm
The conclusion was: Nick to ask the relevant development teams for an estimate of the effort required to implement the automatic removal of entries from the top-level BDII.

*Update Mar 3rd*
Being handled in the ROC managers meeting, closing here.

*Update Feb 1st* : tickets GGUS:32064 (ROC UKI), GGUS:32065 (ROC Russia), GGUS:32067 (ROC DECH), GGUS:32068 (ROC AP), GGUS:32070 (ROC France) submitted to track the issue

*Update Feb 13rd* :
GGUS:32064 (UKI) --> in progress
GGUS:32065 (ROC Russia) --> open. don't allow of queryconf
GGUS:32067 (ROC DECH) --> in progress
GGUS:32068 (ROC AP) --> solved
GGUS:32070 (ROC France) --> child tickets to sites GGUS:32071, GGUS:37072
- GGUS:32072 waiting for reply from Atlas with list of addresses

*Update Feb 18th*:
Insure instructions reaches sites about publishing storage... Lots of tickets submitted, close the item.

*Update Feb 11th* :production sites seem in general not available to provide what requested. The GGUS ticket GGUS:31800 has been set to 'unsolved' and the the issue is being tracked with
http://lblogbook.cern.ch/CCRC08/38

*15th Feb:* broadcast sent

*Update Feb 18th*: Request atlas sites to upgrade WN. Broadcast sent , leave open for a bit, deadline was the 15th March. Review 2 weeks before this.

*Update Mar 3rd* Steve to produce data of queues by OS.
http://straylen.web.cern.ch/straylen/tmp/atlas-gluece-by-os.txt

*Update Mar 10th* From Steve:
http://straylen.web.cern.ch/straylen/tmp/atlas-sites-by-os.txt

*Update Mar 12th* Steve to create a finer report preferable by ROC, ... ( if only that were possible. Maybe via SAM DB)

*Update Mar 19th* Reminder to all sites, time is running out...

*Update Mar 31st* From ATLAS (Alessandro): we have developped a SAM test to see which version of lcg-utils has been installed on the WN of the ATLAS supporting sites. The results can be seen in the sam web page, selecting ATLAS VO, CE, CE-sft-lcg-version. The sites that give ERROR in this test didn't upgrade to the SRM2 compatible version of lcg-utils.
Hope this could help in following the action of having, in all the ATLAS supporting sites, the WN upgraded to SRM2

*Update Apr 21st* A GGUS ticket should be opened against all ROCs to follow-up this issue with sites. Nick knows how to clone a ticket...

*Update May 5th* As soon as ATLAS can confirmed that they've opened a GGUS ticket (cloned for all ROCs), we can close this item.

*Update May 19th* This action can be closed

*15th Feb:* broadcast sent

*18th Feb:* Raised at operations meeting, too soon after broadcast for any feedback.

*19th Mar:* Ongoing, but not obvious how to check compliance.

*31 Mar:* Ongoing. ATLAS will look into building a SAM tests.

*Update Apr 21st* A GGUS ticket should be opened against all ROCs to follow-up this issue with sites. Nick knows how to clone a ticket...

*Update 5th May* Now a GGUS ticket, closing.

*Update 3rd March*
Closed as being tracked by ROC managers.

*Update 12th March*
Received feedback from Italy, Southwest, and a few others. Close action.

*Update: 3/3/08*
Oliver added a "priorities" section to the Node Tracker page:
https://twiki.cern.ch/twiki/bin/view/EGEE/Glite31NodeTracker

Closed.

*Update: 3/3/08*
Announcement should be made at the time of the release of the WMS/LB on SL4 (TBD), saying support will be dropped for the lcg-RB in 2(?) months.

*Update 12 March 2008:* No change.

*Update 31 Mar:* The RB will be obsoleted once the SL4 version of the WMS is available.

*Update 17th Apr:* Will be released in two months, closing.

*Update March 12th*
Had discussion with Ulrich and also submitted a new test for sam, it needs some
thought as to if it is a good idea but it would be a non-critical test on the
"GlueCEStateStatus: Production" attribute that the then critical CE tests would depend on.
The same logic as the existing SE free space tests.
https://savannah.cern.ch/bugs/?34443

*Update 31 March:* Request for new SAM sensor passed to SAM team.

*update 21 April:* On SAM work-list (Savannah). John thought that the item could be closed as far as the ROC managers are concerned, but Kostas was worried that the issue risked being forgotten. He suggested the possibility of a pending state for items that get transferred to other tracking mechanisms. Nick will think about it.

*Update 5th of May* Now on the SAM worklist. Nothing changed for now, ignore for 3 weeks.

*Update 2nd of June* No progress recorded.

*Update 11th June* This is present as BUG:34443 anyway so close here.

Those with old WMS (SL3 in fact)

EENet (Estonia)
ITEP (Russia)
RTUETF ( Latvia)
UNI-FREIBURG (Germany)

Those with new WMS (SL4 in fact)

AEGIS01-PHY-SCL
Australia-ATLAS
BY-UIIP
CERN-PROD
CESGA-EGEE
CGG-LCG2
CNR-PROD-PISA
CY-01-KIMON
CYFRONET-LCG2
DESY-HH
FZK-LCG2
GR-01-AUTH
GRIF
HG-06-EKT
INFN-CNAF
INFN-PADOVA
ITEP
JINR-LCG2
KR-KISTI-GCRT-01
NCP-LCG2
pic
prague_cesnet_lcg2
RAL-LCG2
RO-03-UPB
RTUETF
RU-Phys-SPbSU
ru-PNPI
SARA-MATRIX
Taiwan-LCG2
TR-01-ULAKBIM
UKI-SCOTGRID-GLASGOW
Uniandes
VU-MIF-LCG2

Note there may well be other WMS not included by siteBDIIs out there we know nothing about.

Update 10/9/08: The four sites running WMS on SL3 were asked to upgrade ASAP.

* Answer from Gavin:

The ‘retry’ count is a VO policy, so needs to be set in the relevant VO agent config for the FTS server (the default is 3 retries separated by minimum 10 minutes).

I know CMS’ Phedex prefer to fail-fast (and see the error as early as possible), so have asked T1 sites to set the retry to 0. Phedex then retries externally (i.e. with another FTS jobs for the failed files).

LHCb and ALICE I think are still set to the default.

See: https://twiki.cern.ch/twiki/bin/view/LCG/FtsYaimValues20

Contact fts-support@cernNOSPAMPLEASE.ch is case of problems.

*Update June 11th* Steve should submit tickets to all FTS sites.

*Update June 13th* GGUS:37415 submitted and child tickets sent to ROCs of each Tier1.
Review in two weeks time.

*Update June 20th* GGUS:37415 has been responded to by all FTS instances that
the changes have been made.... Except for:

For USCMS-FNAL-WC1 in GGUS:37428
For BNL-LCG2 in GGUS:37427

Both will be contacted again this week.
* Update June 30th* Steve will escalate, two U.S. sites are problematic.

* Update July 7th* BNL and Fermi have now responded that they made the
configuration change. Action item to be closed after next meeting. Steve

Reviewed 3rd February, Alice = 10 GB, Atlas = 15 GB, CMS = 10GB, LHCb = 2 GB.

All LHC VOs specify values for WN disk space. Close this action, if you see particular
VOs exceeding this then submit GGUS tickets for the VO.

Close this after next operations meeting.

** solved:
LHCb runs a custom SAM test that checks the version of lcg_utils and spots out sites with obsolete version installed.

The person in LHCb following these tickets submitted twice 24 tickets for 24 different sites because his first attempt (using mail ticketing system of GGUS) failed to return the GGUS reference. For your information this problem was due to a missed mapping of the submitters mail address (used by GGUS for submissions of tickets via mail) and his certificate.

*Update 19th March*: Marcin supplied some examples. Problem is well understood, solution is less obvious. John to work with SAM & GridView team.

GGUS:33726

BUG:31940

*Update 13th March 2008*
!BDII2SAM script now fixed, action should be closed following next meeting.

*Update 31 March:* Script is fixed. Close.

GGUS:33850

*update: 20/3/08* GridView team has fixed the bug (CVS tag gridview-synchronizer-20080318).

*update: 7/4/09*: ticket and action re-opened because also gstat needs a fix.

Update 17th April
Min looking at it but the site should really submit a ticket. As per the instructions on
http://goc.grid.sinica.edu.tw/gocwiki/How_to_publish_the_OS_name

Update 17th April
Will be released in at least 2 months, close action item for now.

Update 17th April, Maite will check.

Update 19th May, Andrea changed this on the same day the action was raised. This action can be closed.

Update 17th April, Gilles has done something.

Update 5th May.
CERNROC to provide an update next week once they have been CIC on duty for a
week and cleaned everything up.

Update 9th May.
!YerPhi is now in state suspended and all existing COD tickets have been closed.
This item should be closed after next week's meeting.

Update 19th May, this action can be closed

Update 5th May: Small amount still to do but progress has been made. Revisit next week.

Update 19th May: this action can be closed

*Update Monday June 2nd*
No progress made.

*Update Friday June 6th*
3 pieces of software identified as affected.
lcg-infosites BUG:37572 , lcg-mon-stdout BUG:37571 and glite-sd-query BUG:37569

All are submitted as bugs and this item should be closed after the next monday meeting.
Steve

*Update June 11th* Savanah tickets all submitted, close here.

Update 28-May-08: CMS confirms that the use of pool accounts for SGM has proved to be not working in many cases. The main problem is that the acls on the files are set by users and, if different accounts are used, one software manager could act (e.g. uninstall) packages installed by another.

On the other hand this is not relevant for WMS, where the distinction of pools per VO is not needed.

The conclusion is that the recommended configuration of the accounts has indeed to be different between CEs and WMS, at least as far as CMS is concerned.
As far as I am concerned this action can be closed.

Antonio

*Update 2nd June*
From Romain: There is a SAM test called "CE-wn-sec-crl". General results are public, but detailed results are available only to the ROC security contacts + SAM team.

Follow up question for Romain. CE-wn-sec-crl monitors CRL status on the WNs them self.
What was being asked for was central monitoring of the CA's CRL URLs.... It would make
for an easy rrd plot.

*Update 9th June*
There is central monitoring of CRLs here http://nagios.eugridpma.org/.
Also I have requested that the WLCG Monitoring Group considers getting these
to sites via its alarm/nagios/messaging framework. BUG:37632.

*Update 11th June* Ask the CODS to look at the nagios alarms for CRLs twice
a week.

The reason is that python's tarfile unpacks files with a --preserve-atime so the
files are old as far as tmpwatch is concerned. A way forward is being discussed.

Update 2-Jun: discussion in minutes --> closing

Update 2-Jun: Discussed in minutes --> closing

30/6/08 - Jeremy reckoned this action can be closed.

30/6/08 Site is still failing SAM tests and should be suspended.

3/07/08 Ron reported that the site has now reacted and fixed the situation, close this action after the next operations meeting.

*Update 11th June* Nick will look into this.

*Update 21st June* Waiting for Nick

*Update 28th July* Response from SA3 - _The tarball is produced to work with SL4, so python 2.3 has to be the default. To fully support python 2.5 (for example), you need to distribute the interpreter, reconfigure the environment and, ideally, have all your language extensions recompiled against the new python API. We are looking into how to do the last part, but the first two things are up to the site or VO.

Update 11th August this was raised by a VO in France, is the answer given by SA3 OK? how do we move from here? Helene will pass the feedback to the relevant people
Update 1st Septemberthe action can be closed; finally the real solution was in a savannah bug and it was a problem with the YAIM environment

*Update 11th June* This will be followed up with John and Judit immediately after the meeting on the 9th.

GGUS:36373

July 2nd, got in touch with the job priorities work group to see if they can give a good
example of what should be done.

July 7th, there is now a massive thread which does contain the answer. The answer must
be extracted and documented next.

July16th. I've now written How_to_publish_queues_with_access_restricted_to_a_FQAN but it is unclear
to me what is wanted from the tickets that are assigned to me.

*Update 21/7/08* Now that Wiki page exists, Steve would like to close this item. Any problems should result in new tickets!

Latest version of lcg_utils installed in SAM validation testbed & used against this site.
Previous version failed with: protocol not supported by Storage Element
Latest version fails with: CGSI-gSOAP: Error reading token data header: Connection reset by peer

Problem seems to be with the data that the site provides to the Information System.

*Update 1st July* https://savannah.cern.ch/support/?104871 now submitted.

Steve

Update: GGUS:38050 Problem is that Firefox doesn't recognize the French Certificate Authority. Solution is simply to define an exception in Firefox.

GGUS:37334

2nd July - This may be resolved by a fix to BUG:37008, waiting for clarification.

8th July - Mentioned in the EMT yesterday, the fix is in an upcoming patch and
also a bug will be submitted to link to it. Add bug before next week and close.

14th July - Bug now submitted BUG:38820 . As I understand it this already fixed in
an upcoming release. Close the action here after today's meeting since the BUG is
now present.

GGUS:37890

*14th July 2008* Jeremy will take a look.

*21st July 2008* BUG:38320 which was the related item is now fixed and closed.
This item is, consequently, also closed.

*2nd July* GGUS:38053 now submitted, leave action item until a response is given.

GGUS:34338

Update 8th August. Solved, as was suggested in April the CEs should not present in the GOCDB. GOCDB contains a list of EGEE siteBDIIs and services not under those siteBDIIs are not at the site as far as GOCDB/gstat or EGEE is concerned. Steve.

Close this action after next meeting.

Update 25th August SAMAP will follow-up "later"
Update 8/9/08: Nick will follow-up.
Update 13th October The tool development team has fixed the bug.

Update 25th August: This will be released in the next update to gLite 3.1 - within 1-2 weeks.
1st September After teh update at today's meeting, this action can be closed: the EMT made the decision to delay the deployment of the CREAM CE (the certified patch). This is because not-ICE-enabled WMS could accidentally match the Cream CE and cause a submission failure. Waiting for the ICE-WMS to be deployed, as a workaround, Cream will be released with a GlueServiceStatus?? = ‘Production’, to be changed again later. One issue is represented by the old version of WMS on SL3 (unsupported). As they will not be integrated with ICE, once the Cream CE will be advertised again in real production mode, they would fail to submit. In order to size this issue up we would like to get from the WLCG EGEE Operation Meeting an estimation of the number of old SL3 WMS still in production.

Update August 19th: Maite has some news?

Update 1st September SAM agrees to extend the 7 days period in this specific case: the CA RPMs are not put in the repository in the 1 day scheduled for this. Technically it is feasible and already implemented. See diagram and explanations here:
https://twiki.cern.ch/twiki/bin/view/LCG/SAMSensorsTests#CE_sft_caver

Shorty, the diagram shows that it is possible to configure:
- time-stamp from which countdown of timeout starts
- delay of warning
- timeout before sites will get CRIT error

Update 10/9/08: Although Nick doesn't understand the text, he said that the ticket can be closed (SAM implemented what was asked).

Update 19th August - Slow response no longer obvious, will close and reopen if need be.

GGUS:39906

The action can be closed

BUG:39641

the action can be closed

Was raised at this week's LCG MB. Fermilab representatives are checking internally if it is still needed.

Update 19th of August: The LCB MB meets today and this will hopefully be resolved.

Update 25th August: The KCA will soon be officially approved as a trusted CA. Also, it is being used by the CDF VO. Therefore, KCA will remain in the list of CAs.

Update 10/9/08: Nick sent an EGEE broadcast about this. In fact, he sent two, to explain that this only concerned sites supporting VOs that had explicitly mentioned the shared software area in their respective VO cards.

GGUS:42341

Update: Current monitoring architecture does not allow one service to be on multiple nodes/sites. A new database schema is being worked on (based on service end-points), but the restriction of binding a service to a specific site will probbaly remain.

GGUS:42015

*Update 17th November*, site has now responded.
North East ROC to respond next week.

*Update 27th November*, site has now corrected the problem, closing this.

Update: link was always there, but now uses a font for the blind.

https://gus.fzk.de/ws/ticket_info.php?ticket=43994

Update 19/1/2009: Biomed GFAL version problem, Maite will send broadcast after the meeting (seems some sites are still on SL3 and need to upgrade the O/S as well as GFAL!)

Update 26/1/2009: no broadcast seen, OCC to follow up.

Update 2/2/2009: broadcast not sent, problem being followed up with sites through GGUS. Agreement to close item.

Update 19/1/2009: (mail from Akos):
We have examined the issue and it does not look like a security problem, but a resource limitation: the number of threads in an LFC instance limits the number of clients that can connect concurrently and the Biomed usage patter exceeds that limit.
When the clients would finish their work, LFC would be responsive again.

The same problem would occur with other iterator like operations, like opendir/readdir/closedir.

Removing these operations would cause old clients to fail, however it would not solve the problem, so in my opinion the upgrade of lcg_utils is the right solution.

Unfortunately nobody has contacted us from the Biomed community regarding the possibility and context of a special build, so we did not progress on that side.

Update 26/1/2009: Can be closed.

Update 19/1/2009: Long-term solution could be SAM tests, or adding GFAL version collection to job-wrapper scripts.

Update: 19/1/2009:
The outcome of the get-together was:

>> Not having SE affects on passing by site RM SAM tests - those tests take closest SE (default).
This is incorrect – the defined SE doesn’t have to be at the site!

>> Also setting up site in such situation is not possible because yaim require SE.
Correct, but again the SE doesn’t have to be local to the site.

>> In case of putting SE in Scheduled downtime, site have to put also CE into downtime (otherwise will not pass RM tests) or chose (lack in procedures) other SE (from other site).

This is correct, and the only real issue. ATLAS doen’t use Replica Management tests, but believe that they should be part of the ops infrastructure tests (which are more extensive). There may be a case for making the replica management tests non-critical, but they’ve been critical for two years now and most people seem happy with this.

The way for a site to change the defined SE is to modify the variable VO_OPS_DEFAULT_SE in the WNs’ site-info.def files.

Update 19/1/2009: CERN ROC to check sites with only core services – no progress.

Update 2/2/2009: New SWE site RedIRIS will only host core services (BDII, WMS, etc.)

Problems until now:

1) GIIS performance error due to: GIIS Old Entries Found: 6 - ERROR
- This will make the SAM test gperf fail.

2) No Grid Version published: GridVersion: *NOTE* could not find valid LCG version
- This ist just a warning in GSTAT at this moment

The other tests seem to work only the gperf error is critical.

Update 12th February - Steve will take a look to understand what this is about.

Update 19th February - Steve - Confused , there is no RedIRIS site in gstat? http://gstat.gridops.org/gstat//SouthWesternEurope.html

Update at the meeting - Kai will check.

Update 3rd March The gstat errors are caused by the WMS publishing only static information. The new info provider just release publishes dynamic information
so this will fix itsef.

*Update 12th Feb* - There is no list. There is a list of per VO on the VO Cards, we may try and produce a common list. What next?

*Update at the meeting* - VOs will definitely have to maintain a list of the libraries they need, in their VO ID card.
Item closed.

12th Feb, TWiki page has been created.

GGUS:45038

Feb 23rd Ticket is now solved, site in quarantine. Close this action item after next operations meeting.

Update at the meeting - Confusion as to what this action was about. Whether Condor is supported on the gLite 3.1 LCG CE. Nick will follow up.

Update 27 Feb 09 - In theory the gLite 3.1 LCG CE should support the Condor batch system. Instructions on how to set it up are here: https://twiki.cern.ch/twiki/bin/view/EGEE/BatchSystems. If a site has problems, please submit a GGUS ticket and CC neissner@picNOSPAMPLEASE.es.

Update 9 Mar 09 - closing

Update 27/2/09: It turns out that GridView does not synchronize on that particular GOCDB field, so it isn't available to SAM. The recommended workaround is to create a scheduled downtime - tests will still run, but no tickets will be raised. The requested functionality will be in the new Aggregated Topology Provider, and GOCDB will have a production attribute associated with each service.

Update 9/3/09 - closing during meeting

PATCH:2784

UPDATE 20-Mar: I discussed with the EMT Coordinator. There is a long list of services which now have priority both in certification and release preparation. They welcomed the idea of a pilot service to be run at some sites, which by providing real-usage records would help them making the certification faster. I am preparing a request for sites to join this activity to be presented at the next OPS meeting.

Update 23/3/09: other than some SAM alarms due to temporary glitches with the central LFC and Top-level BDII, no problems were noted.

Last meeting agreed to close this item, but it was not done at the time.

UPDATE 25/3/2009
Two GGUS tickets have been open
for CERN 47367
for FNAL 47368

Update 20/4/09: This should have been closed last week; issue being tracked in GGUS.

Update 20/4/09: Nick to check with Gilles (but he thinks that the answer is no).

Update 27/4/09: An update has been given to Nick by Gilles, this will be added here.

18/05/09: An RSS notification is sent by the Operations Portal whenever there is a change to a down-time (see minutes of today's meeting for more details).

Update 27th April. A timetable will be provided by OSCT.

Update 4th May: No impact. OSCT will provide a time table, carrying on the old schedule from the COD to be applied to the OSCT till teh end of EGEE III.

31st August - ROC should go ahead and certify the site once happy. Its a valid configuration and any problems in the monitoring or similar should be fixed.
At the moment none are expected.

This wasn't done fast enough to be useful, so closing the action. John 25/8/09