(r108) GLUEMonitoring < EGEE

EGEE Web>InformationSystem>GLUEMonitoring (revision 108) (raw view)~~EditAttachPDF~~
---+ !GLUE2 publication monitoring

This twiki tracks the effort of monitoring the quality of the GLUE 2 information published in the WLCG information system. Ongoing efforts are already looking into the information published by the [[https://twiki.cern.ch/twiki/bin/view/EGEE/ISproviders#GLUE_2_0_Validation][information providers]] to detect bugs in the middleware that should be fixed by the middleware developers.

This twiki collects the monthly reports that evaluate the results of running glue-validator against WLCG sites.
%TOC%

---++ Reports

   * [[https://twiki.cern.ch/twiki/pub/EGEE/GLUEMonitoring/WLCG-GLUE2-validation-statistics-July2013.pdf][July 2013]]
   * [[https://twiki.cern.ch/twiki/pub/EGEE/GLUEMonitoring/WLCG-GLUE2-validation-statistics-June2013.pdf][June 2013]]
   * [[https://twiki.cern.ch/twiki/pub/EGEE/GLUEMonitoring/WLCG-GLUE2-validation-statistics-May2013.pdf][May 2013]]


---++ Known Issues

---+++ Bug deleting GLUE 2 entries

As described in BUG:101237, the !BDII fails to delete old GLUE 2 entries due to a bug in the code. This causes a pollution of GLUE 2.0 obsolete objects in the Information System. A workaround to this problem is to restart all resource !BDIIs in the site, then remove the contents of =/var/lib/bdii/gip/cache/gip/site-urls.conf-glue2= in the site !BDII and restart the site !BDII. Due to this bug, the glue-validator raises many [[https://twiki.cern.ch/twiki/bin/view/EGEE/GLUEValidatorErrorCodes][E002]] errors.

---+++ Misconfiguration of load balanced services

It has been noted that sites fail to properly declare load balanced services in the information system. All machines behind an alias, must be declared as site resources using the machine hostname. For example, using YAIM for a load balanced site !BDII:

<verbatim>
BDII_REGIONS="CE SE TOPBDII SITEBDII_1 SITEBDII_2"

...
SITEBDII_BDII_1_URL="ldap://<bdii1-hostname>:2170/mds-vo-name=resource,o=grid"
SITEBDII_BDII_2_URL="ldap://<bdii2-hostname>:2170/mds-vo-name=resource,o=grid"

SITE_BDII_HOSTNAME=<service-alias>
</verbatim>

This means that we should see the !GLUE2ServiceID and !GLUE2EndpointID with the real hostnames of the machines and the !GLUE2EndpointURL with the DNS alias. 

---+++ Failed hostname -f

When the command =hostname -f= fails to be executed in a machine, !GLUE2ServiceID and !GLUE2EndpointID are wrongly generated. It has been observed that in most cases this is a temporary failure and then the command works again and the IDs can be properly generated. Due to BUG:101237, these wrong entries stay in the system and are never deleted. The command below shows this problem for site !BDIIs:

<verbatim>
ldapsearch -LLL -x -h lcg-bdii -p 2170 -b GLUE2GroupID=grid,o=glue '(&(objectClass=GLUE2Endpoint)(GLUE2EndpointID=_bdii-site_3536672524_bdii_site_3536672524))'  dn | grep GLUE2DomainID | cut -d"=" -f3 | cut -d"," -f1
</verbatim>

This issue was reported in BUG:101562 and a fix has been released in EMI 2 and EMI 3.

---+++ Huge amount of information published by !GLUE2ApplicationEnvironment

MAY-22-2013:

In GLUE 2.0, we are currently publishing ~195.000 DNs:

<verbatim>
ldapsearch -LLL -x -h lcg-bdii -p 2170 -b o=glue dn | grep dn: | wc -l
194834
</verbatim>

In GLUE 1.3, we are currently publishing ~68.000 DNs:
<verbatim>
ldapsearch -LLL -x -h lcg-bdii -p 2170 -b o=grid dn | grep dn: | wc -l
67760
</verbatim>

When trying to understand this difference, we have realized that GLUE 2 is publishing ~ 135.000 !GLUE2ApplicationEnvironment DNs:

<verbatim>
ldapsearch -LLL -x -h lcg-bdii -p 2170 -b o=glue '(objectClass=GLUE2ApplicationEnvironment)' dn | grep dn: | wc -l
134987
</verbatim>
 
These objects sum up ~120MB of information when the real information is actually contained in one attribute, !GLUE2ApplicationEnvironmentAppName. Collecting only this information is 11MB:

<verbatim>
ldapsearch -LLL -x -h lcg-bdii -p 2170 -b o=glue '(objectClass=GLUE2ApplicationEnvironment)' GLUE2ApplicationEnvironmentAppName | grep GLUE2ApplicationEnvironmentAppName: > appname
ldapsearch -LLL -x -h lcg-bdii -p 2170 -b o=glue '(objectClass=GLUE2ApplicationEnvironment)' > appenv

-rw-r--r--. 1 root root 120M May 22 09:54 appenv
-rw-r--r--. 1 root root  11M May 22 09:53 appname
</verbatim>

It should also be noted that due to BUG:101237, there are ~18.000 obsolete !GLUE2ApplicationEnvironment objects, out of which ~1.700 objects are not publishing at all the creation time:

<verbatim>
ldapsearch -LLL -x -h lcg-bdii -p 2170 -b o=glue '(&(objectClass=GLUE2ApplicationEnvironment)(!(GLUE2EntityCreationTime=2013-05-22*)))' dn | grep dn: | wc -l
17778

ldapsearch -LLL -x -h lcg-bdii -p 2170 -b o=glue '(&(objectClass=GLUE2ApplicationEnvironment)(!(GLUE2EntityCreationTime=*)))' dn | grep dn: | wc -l
1665
</verbatim>

---+++ !GluePolicy/GLUE2ComputingShare attributes

The following table explains how the !GluePolicy and !GLUE2ComputingShare attributes are defined by the information providers.
Note that there is a more detailed table for PBS available in the [[https://wiki.italiangrid.it/twiki/bin/view/CREAM/SystemAdministratorGuideForEMI2#1_6_1_3_Manual_Tuning][CREAM Sys Admin guide]].

| Glue 1.3 attribute | GLUE 2.0 attribute | PBS/Torque queue attribute | LSF queue attribute | SGE queue attribute |
| !GluePolicyMaxWallClockTime | !GLUE2ComputingShareDefaultWallTime (*) | =resources_default.walltime= if defined, otherwise =resources_max.walltime= (seconds, or [[HH:]MM:]SS) | =RUNLIMIT=  (hours:minutes or minutes) | cpu  |
| !GluePolicyMaxObtainableWallClockTime |  !GLUE2ComputingShareMaxWallTime (*) | =resources_max.walltime= (seconds, or [[HH:]MM:]SS) | =RUNLIMIT=  (hours:minutes or minutes) | h_rt  |
| !GluePolicyMaxCPUTime | !GLUE2ComputingShareDefaultCPUTime (*) | =min(resources_default.cput, resources_default.pcput)= if defined, =min(resources_max.cput, resources_max.pcput)= otherwise (seconds, or [[HH:]MM:]SS) | =CPULIMIT=  (hours:minutes or minutes) | cpu |
| !GluePolicyMaxObtainableCPUTime | !GLUE2ComputingShareMaxCPUTime (*) | =min(resources_max.cput, resources_max.pcput)= (seconds, or [[HH:]MM:]SS) | =CPULIMIT=  (hours:minutes or minutes) | h_rt |
| !GluePolicyMaxTotalJobs | !GLUE2ComputingShareMaxTotalJobs | =max_queuable= | Complex computation | - |
| !GluePolicyMaxRunningJobs | !GLUE2ComputingShareMaxRunningJobs | =max_running= | Complex computation | - |
| !GluePolicyMaxWaitingJobs | !GLUE2ComputingShareMaxWaitingJobs | =max_queueable - max_running= | Complex computation | - |
| !GluePolicyMaxSlotsPerJob | !GLUE2ComputingShareMaxSlotsPerJob | =resources_default.procct= if defined, else =resources_max.procct= | Complex computation | - |
| !GluePolicyAssignedJobSlots | !GLUE2ComputingShareAssignedJobSlots | =np= (from pbsnodes -a -s) | Complex computation | - |

   * For PBS/Torque, the queue configuration is retrieved using =qstat -Q -f= in most cases. 
      * See [[https://github.com/italiangrid/info-dynamic-scheduler-pbs][lcg-info-dynamic-scheduler-pbs]] code for more details. 
      * See [[http://docs.adaptivecomputing.com/torque/4-0-2/Content/topics/4-serverPolicies/queueAttributes.htm][PBS queue attributes]] documentation.
   * For LSF, the queue configuration is retrieved using =bqueues -l= in most cases but in order to calculate the values, a complex computation is needed in many cases due to the nature of LSF. 
      * See [[https://github.com/schwicke/info.dynamic-scheduler-lsf][info-dynamic-scheduler-lsf]] code for more details.
      * See [[http://www.cisl.ucar.edu/docs/LSF/7.0.3/configuration_reference/index.htm?lsb.queues.5.html~main][LSF queue attributes]] documentation.
   * For SGE, the queue configuration is retrieved using =qconf -sq opsgrid | egrep "(h_rt|h_cpu)"= and the results are transformed in seconds by the info provider.
   * (*)Known Issue: Time is published in hours in GLUE 1.3 (should be minutes) and in minutes in GLUE 2 (should be seconds). See this BUG:101076 and the documented [[https://wiki.italiangrid.it/twiki/bin/view/CREAM/KnownIssues#Wrong_time_format_for_MaxWallClo][CREAM Known Issue]] for more details.

---+++ GLUE 1 and GLUE 2 mismatch

The following !YAIM variables are used for both GLUE 1 and GLUE 2:

   * =CE_LOGCPU=: Total number of cores/hyperthreaded CPUs in the !SubCluster
   * =CE_PHYSCPU=: Total number of real CPUs/physical chips in the !SubCluster

However the definitions in GLUE 2 are:

   * !GLUE2ExecutionEnvironmentLogicalCPUs: The number of logical CPUs in one Execution Environment instance, i.e. typically the number of cores per Worker Node
   * !GLUE2ExecutionEnvironmentPhysicalCPUs: The number of physical CPUs in one !ExecutionEnvironment instance, i.e. the number of sockets per Worker Node

This means that in GLUE 2 we are currently publishing wrong information according to the definition of these variables. This has been tracked by the CREAM developers on his list of [[https://issues.infn.it/jira/browse/CREAM-106?focusedCommentId=38960&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-38960][known issues]].

---++ Field Work

The following tables track tickets opened to sites to follow up on incorrect values published in the information system. Some of these issues are related to bugs in the information providers and some of them are due to misconfigurations in the sites. The tables below try to summarise the findings related to wrong storage and computing information to find common patterns in case of misconfigurations and to make sure information providers are fixed when a bug is found.

Apart from the BDII, the following sources of information have been used:
   * [[http://dashb-ssb.cern.ch/dashboard/request.py/siteview#currentView=storage&highlight=true][CMS Site Status Board]] - [[https://twiki.cern.ch/twiki/bin/view/CMSPublic/MonStorageSSB][Documentation]] (Check CMS sites names to WLCG site names [[https://cmsweb.cern.ch/sitedb/reports/showXMLReport/?reportid=naming_convention.ini][translation]])
   * [[http://bourricot.cern.ch/dq2/accounting/bdii_vs_srm/][ATLAS storage information: SRM vs BDII]]
   * [[http://lhcbproject.web.cern.ch/lhcbproject/Operations/queues.html][LHCb queues]]

---+++ General Storage

| GGUS ticket | Summary | Cause | Affected Service | Affected GLUE Attributes | Affected site |
| GGUS:87570 | Stange space values | Not known yet. No reaction from the site | !DPM | Unknown | TR-03-METU |
| GGUS:90219 | Negative used space | Not known yet. Seems to be a bug in the !DPM Information Providers | !DPM | !GlueSAStateUsedSpace: -3987365412 | UKI-LT2-RHUL |
| GGUS:90319 | Strange space values | Investigations are ongoing. Seems to be caused by an upgrade of the information providers (old static ldfil file) | dCache | !GlueSAUsedOnlineSize: 0<br>GlueSAStateUsedSpace: 999999<br>GlueSAReservedOnlineSize: 0<br>GlueSATotalOnlineSize: 0 | INFN-ROMA1-CMS |
| GGUS:90321 | Negative free space | Not known yet. No reaction from the site | !StoRM | !GlueSAFreeOnlineSize | INFN-T1 |
| GGUS:90325 | Strange space values | Now known yet. No reaction from the site | !StoRM | !GlueSETotalOnlineSize: 51147657<br>GlueSEUsedOnlineSize: 0<br>GlueSAStateUsedSpace: 0<br>GlueSATotalOnlineSize: 1400000| INFN-BARI |
| GGUS:90328 | Strange space values | Not known yet. Seems to be a bug in !StoRM | !StoRM | !GlueSETotalOnlineSize: 105011<br>GlueSEUsedOnlineSize: 0 | UKI-SOUTHGRID-BRIS-HEP |

---+++ General Computing

| GGUS ticket | Summary | Cause | Affected Service | Affected GLUE Attributes | Affected site |
| GGUS:88754 | Empty Value | Not specified but fixed by the site | CREAM PBS | !GlueCEPolicyMaxCPUTime: 0<br>GlueCEPolicyMaxWallClockTime: 0 | !IN2P3-CPPM |
| GGUS:88772 | 999999999 Value | Due to GGUS:82902. For LSF it will be fixed in EMI 2 Update 8 scheduled for end January 2013 | CREAM LSF | !GlueCEPolicyMaxCPUTime: 999999999 | UKI-NORTHGRID-LANCS-HEP |
| GGUS:88773 | 999999999 Value | Due to GGUS:82902. For LSF it will be fixed in EMI 2 Update 8 scheduled for end January 2013 | CREAM LSF | !GlueCEPolicyMaxCPUTime: 9999999999 | INFN-PISA |
| GGUS:88781 | 99999999 Value | Site manually setting !MaxCPUTime=MaxWallClockTime. Fixed after LHCb requested it. | CREAM LSF | !GlueCEPolicyMaxCPUTime: 999999999 | UKI-NORTHGRID-SHEF-HEP |
| GGUS:88822 | 999999999 Value | Fixed manually but also suffering from GGUS:82902 | CREAM SGE | !GlueCEPolicyMaxCPUTime: 999999999 | UKI-LT2-QMUL |
| GGUS:89847 | Unexpected value | Not known yet. No reaction from the site. | Unknown | Unknown | RO-07-NIPNE |
| GGUS:89857 | 999999999 Value | Fixed manually but also suffering from GGUS:82902 | CREAM SGE | !GlueCEPolicyMaxCPUTime: 999999999 | FZK-LCG2 |

---+++ Site !BDII not published as part of the site

| Site | GGUS ticket | Comments | 
| IN-DAE-VECC-02 | GGUS:93809 | %Y% |
| INDIACMS-TIFR | GGUS:93808 | %Y% site !BDII now published as part of the site. Stop publishing top !BDII as part of the site (in fact publishing CERN top !BDII) |
| NO-NORGRID-T2 | GGUS:93810  | %Y% |
| praguelcg2 | GGUS:93197 | %Y% Publishing CERN top level !BDII as the site top level !BDII |
| SE-SNIC-T2 | GGUS:93835  | %Y% |
| T2_Estonia | GGUS:93801 | %Y% |
| UKI-LT2-IC-HEP | GGUS:94096 | %Y% |
| ru-Moscow-SINP-LCG2 | GGUS:99498 | %X% |

---+++ !GLUE2ComputingShareMaxCPUTime

Campaign to fix LHCb sites that are publishing the default 9999999 in the [[http://lhcbproject.web.cern.ch/lhcbproject/Operations/queues.html][LHCb queues dashboard]]. Note that this is a recurring problem and this is now dynamically tracked in the [[https://wlcg-mon/dashboard/request.py/siteview#currentView=Glue+Validator+LHCb][Monitoring Dashboard]].

Also note that as requested in GGUS:97721, CREAM developers will implement a change in the info provider to distinguished unlimited from undefined values.

| Site | GGUS ticket | Comments |
| BG01-IPP | GGUS:94621 | %Y% OK after fixing configuration error in the batch system configuration |
| GRISU-UNINA | GGUS:94718 | %Y% OK after fixing configuration error. Missing resources_max.cput and resources_default.cput  |
| INFN-T1 | - | %Y% Published value in !BDII is correct: wrong value in the LHCb dashboard is now up to date |
| INFN-TRIESTE | GGUS:94554 | %Y% LHCb pointing to wrong queue. The correct queue publishes non default values |
| RUG-CIT | - | %Y% Published value in !BDII is correct: wrong value in the LHCb dashboard is now up to date |
| SARA-MATRIX | GGUS:94619 | %Y% No limits. For LHCb limit has been configured as requested: LHCb dashboard was pointing to a queue that was in fact not supported for LHCb |
| UA-KNU | GGUS:94720 | %Y% OK after fixing configuration error. Missing resources_max.cput and resources_default.cput: LHCb dashboard has removed this queue since in fact it is not supported for LHCb |
| UKI-LT2-IC-HEP | GGUS:95315 | %Y% LHCb pointing to the wrong queue. The correct queue publishes non default values | 
| UKI-LT2-QMUL | GGUS:94510 | %Y% OK after upgrading to EMI 3 |
| UKI-NORTHGRID-SHEF-HEP | GGUS:94618 | %Y% OK after fixing configuration error. Missing resources_max.cput and resources_default.cput |
| UNI-DORTMUND | GGUS:94717 | %Y% Testing CE published as 'Production' used by LHCb. The production CE publishes correct Max CPU times |
| UNINA-EGEE | GGUS:94719 | %Y% OK after fixing configuration error. Missing resources_max.cput and resources_default.cput |
| RU-SPbSU | GGUS:94620 | %Y% OK after fixing configuration error in =/etc/lrms/pbs.conf=. Hostname was not defined |


---+++ Operating System information

Campaign to get LHCb sites to publish coherent OS name, version and releases. The relevant GLUE attributes are:

| GLUE 1.3 | GLUE 2.0 |
| !GlueHostOperatingSystemRelease | !GLUE2ExecutionEnvironmentOSVersion |
| !GlueHostOperatingSystemName | !GLUE2ExecutionEnvironmentOSName |
| !GlueHostOperatingSystemVersion | NA |
| NA | [[http://glue20.web.cern.ch/glue20/#b23][GLUE2ExecutionEnvironmentOSFamily]] |

For SL, the following =version - release= has to be respected:
   * SL 4 series - Beryllium
   * SL 5 series - Boron
   * SL 6 series - Carbon

The instructions on how to publish OS information has been described by EGI in the [[https://wiki.egi.eu/wiki/HOWTO05_How_to_publish_the_OS_name][HOWTO05]] manual.

| Site | GGUS ticket | comments |
| BMEGrid | GGUS:94840 | %Y% Fixed wrong OS Name + version |
| CY-01-KIMON | GGUS:94854 | %Y% Fixed inconsistent OS release + Version |
| GRISU-UNINA | GGUS:94855 | %Y% Fixed inconsistent OS release + Version |
| IFJ-PAN-BG | GGUS:94856 | %Y% Fixed inconsistent OS release + Version |
| INFN-CATANIA | GGUS:94857 | %Y% Fixed inconsistent OS release + Version | 
| INFN-FERRARA | GGUS:94841 | %Y% Fixed wrong OS Name + version |
| INFN-NAPOLI-ATLAS | GGUS:94842 and GGUS:94858 | %Y% Wrong OS Name + version and Inconsistent OS release + Version |
| INSU01-PARIS | GGUS:94859 | %X% release is wrong | 
| PSNC | GGUS:94860 | %Y% Fixed inconsistent OS release + Version | 
| RAL-LCG2 | GGUS:94861 | %Y% Fixed inconsistent OS release + Version | 
| RO-07-NIPNE | GGUS:94862 | %Y% Inconsistent OS release + Version | 
| RO-11-NIPNE | GGUS:94864 | %Y% Fixed inconsistent OS release + Version | 
| RO-15-NIPNE | GGUS:94865 | %Y% Fixed inconsistent OS release + Version | 
| RU-SPbSU | GGUS:94844  | %Y% Wrong OS Name + version |
| Ru-Troitsk-INR-LCG2 | GGUS:94866 | %Y% Fixed inconsistent OS release + Version | 
| TECHNION-HEP | GGUS:94867 | %Y% Fixed inconsistent OS release + Version | 
| UA-KNU | GGUS:94845 | %Y% Fixed wrong OS Name |
| UKI-LT2-Brunel | GGUS:94868 | %Y% Fixed inconsistent OS release + Version | 
| UKI-LT2-QMUL | GGUS:94869 | %Y% Fixed inconsistent OS release + Version | 
| UKI-NORTHGRID-LANCS-HEP | GGUS:94870 | %Y% Fixed inconsistent OS release + Version | 
| UKI-NORTHGRID-MAN-HEP | GGUS:94871 | %Y% Fixed inconsistent OS release + Version | 
| UKI-SCOTGRID-ECDF | GGUS:94873 | %Y% Inconsistent OS release + Version | 
| UKI-SOUTHGRID-BHAM-HEP | GGUS:94879 | %Y% Fixed wrong OS release |
| UNINA-EGEE | GGUS:94874 | %Y% Fixed inconsistent OS release + Version | 

---+++ Storage Service and Share Capacity

---++++ !TotalSize <> !ReservedSize + !FreeSize + !UsedSize

Some storage services are publishing storage capacity size attributes in such a way that =TotalSize=ReservedSize+FreeSize+UsedSize= will never match. The glue-validator has applied the following workarounds taking into account each storage service:
   * All: even if some attribute is missing, the calculation is done with the published attributes.
   * !DPM: =ReservedSize= is always equal to =TotalSize= in space tokens. Therefore, =ReservedSize= is not used in the calculation.
   * dCache: For online capacity, the numbers always match although =ReservedSize= is not published. For nearline capacity, =FreeSize= is not published, so it is not possible to make the numbers match (although this number is not published because =Free=Total-Used=).
   * !StoRM: For online capacity, the numbers always match. There are some issues with nearline capacity. In general, there is no need for any workaround in the !StoRM case.

| Site | GGUS ticket | comments |
| pic | GGUS:95668 | %Y% dCache. pic confirms numbers > 1 million GB (*) or < 1000 GB are correct. |
| UKI-SCOTGRID-GLASGOW | GGUS:95816 | %Y% !DPM. OK after understanding how !DPM is calculating numbers |
| UKI-SCOTGRID-ECDF | GGUS:95817 | %Y% !DPM. OK after understanding how !DPM is calculating numbers |
| praguelcg2| GGUS:96326 | %X% !DPM: wrong numbers for unreserved space, which is a known DPM issue |
| INFN-T1 | GGUS:95665 | %X% !StoRM. This is a known StoRM issue tracked in GGUS:95666 |
| RUG-CIT | GGUS:95666 | %X% !StoRM. Values > 1 million GB (*) are correct. Nearline storage seems to be a problem. See GGUS:95666 |
| TR-03-METU | GGUS:95667 | %Y% !DPM. Wrong service capacity numbers have been fixed |

(*) Note that glue-validator was checking whether the storage capacity was higher than 1 million GB instead of 1 billion GB.

---++++ Capacity > 1 billion GB

| Site | GGUS ticket | comments |
| INFN-PISA | GGUS:96486 | %Y% incorrect values in the YAIM variable | 
| CSCS-LCG2 | GGUS:96489 | %Y% Site thought values should be published in bytes instead of GB  |
| UNI-FREIBURG | GGUS:96490 | %Y% Fixed value in  /etc/dcache/info-provider.xml |


---+++ 444444 waiting jobs 

Check the following [[https://wiki.egi.eu/wiki/Tools/Manuals/TS59][EGI manual]] for more details. The table below contains the list of batch system related GLUE attributes that are dynamically modified by the batch system information providers, and whether they are known to be properly calculated and actually updated by the information providers:

| Info Provider | GLUE 1 attribute | GLUE 2 attribute | Status |
| [[https://github.com/italiangrid/info-dynamic-scheduler-generic][info dynamic scheduler]] | !GlueCEStateTotalJobs | !GLUE2ComputingShareTotalJobs |
|^| !GlueCEStateRunningJobs | !GLUE2ComputingShareRunningJobs |
|^| !GlueCEStateWaitingJobs | !GLUE2ComputingShareWaitingJobs |
|^| !GlueCEStateEstimatedResponseTime | !GLUE2ComputingShareEstimatedAverageWaitingTime |
|^| !GlueCEStateWorstResponseTime | !GLUE2ComputingShareEstimatedWorstWaitingTime |
|^| !GlueCEStateFreeJobSlots | !GLUE2ComputingShareFreeSlots |
|^| NA | !GLUE2ComputingShareUsedSlots  | Not published |
| [[https://github.com/schwicke/info.dynamic-scheduler-lsf][info dynamic LSF]] | !GlueCEPolicyMaxObtainableCPUTime | !GLUE2ComputingShareDefaultCPUTime |
|^| !GlueCEPolicyMaxCPUTime | !GLUE2ComputingShareMaxCPUTime |
|^| !GlueCEPolicyMaxObtainableWallClockTime | !GLUE2ComputingShareDefaultWallTime |
|^| !GlueCEPolicyMaxWallClockTime | !GLUE2ComputingShareMaxWallTime |
|^| !GlueCEPolicyMaxTotalJobs | !GLUE2ComputingShareMaxTotalJobs |  
|^| !GlueCEPolicyMaxRunningJobs | !GLUE2ComputingShareMaxRunningJobs |
|^| !GlueCEPolicyMaxWaitingJobs | !GLUE2ComputingShareMaxWaitingJobs | Calculated in v 2.3.1-3 |
|^| !GlueCEPolicyMaxSlotsPerJob | !GLUE2ComputingShareMaxSlotsPerJob | Calculated in v 2.3.1-3 |
|^| NA | !GLUE2ComputingShareMaxMainMemory | Calculated in v 2.3.1-3 |
|^| NA | !GLUE2ComputingShareMaxVirtualMemory | Calculated in v 2.3.1-3 |
| [[https://github.com/italiangrid/info-dynamic-scheduler-pbs][info dynamic PBS]] | !GlueCEPolicyMaxObtainableCPUTime | !GLUE2ComputingShareDefaultCPUTime |
|^| !GlueCEPolicyMaxCPUTime | !GLUE2ComputingShareMaxCPUTime |
|^| !GlueCEPolicyMaxObtainableWallClockTime | !GLUE2ComputingShareDefaultWallTime |
|^| !GlueCEPolicyMaxWallClockTime | !GLUE2ComputingShareMaxWallTime |
|^| !GlueCEPolicyMaxTotalJobs | !GLUE2ComputingShareMaxTotalJobs | 
|^| !GlueCEPolicyMaxRunningJobs | !GLUE2ComputingShareMaxRunningJobs |
|^| !GlueCEPolicyMaxWaitingJobs | !GLUE2ComputingShareMaxWaitingJobs |
|^| !GlueCEPolicyMaxSlotsPerJob | !GLUE2ComputingShareMaxSlotsPerJob | Calculated in v. 2.3.2-2 |
|^| NA | !GLUE2ComputingShareMaxMainMemory |
|^| NA | !GLUE2ComputingShareMaxVirtualMemory |
| [[https://github.com/ge-utils/info-dynamic-scheduler][info dynamic SGE]] |  !GlueCEPolicyMaxObtainableCPUTime | !GLUE2ComputingShareDefaultCPUTime |
|^| !GlueCEPolicyMaxCPUTime | !GLUE2ComputingShareMaxCPUTime |
|^| !GlueCEPolicyMaxObtainableWallClockTime | !GLUE2ComputingShareDefaultWallTime |
|^| !GlueCEPolicyMaxWallClockTime | !GLUE2ComputingShareMaxWallTime |
|^| !GlueCEPolicyMaxTotalJobs | !GLUE2ComputingShareMaxTotalJobs | 
|^| !GlueCEPolicyMaxRunningJobs | !GLUE2ComputingShareMaxRunningJobs |
|^| !GlueCEPolicyMaxWaitingJobs | !GLUE2ComputingShareMaxWaitingJobs |
|^| !GlueCEPolicyMaxSlotsPerJob | !GLUE2ComputingShareMaxSlotsPerJob | |
|^| NA | !GLUE2ComputingShareMaxMainMemory |
|^| NA | !GLUE2ComputingShareMaxVirtualMemory |

| SITE | GGUS ticket | comments |
| CERN-PROD | GGUS:96529 | %Y% Some variables were not published by the LSF info provider. This has been fixed in the code for a future release and directly in the production CEs. |
| UKI-SCOTGRID-DURHAM | GGUS:96530 | %Y% This was a transient error, it's fixed now  |
| UKI-SCOTGRID-GLASGOW | GGUS:96528 | %Y% The site was missing the directory where ERT/WRT calculations are stored. This is normally created by !YAIM, but for some reason this wasn't created. After manual creation, it published proper values. |
| UKI-SOUTHGRID-RALPP | GGUS:96531 | %Y% lcg-info-dynamic-scheduler-pbs was not installed in the CLUSTER node that was running on a different host than the CREAM CE. CREAM developers will make sure this is properly documented in the CLUSTER installation notes.  |

---+++ Publishing Domain called 'resource'

This problem seems to be related to a bug in a quattor template.

| SITE | GGUS ticket | comments |
| BEgrid-ULB-VUB | GGUS:18121 | %Y% |
| BEIJING-LCG2 | GGUS:98120 | %Y% |
| !IN2P3-CPPM | GGUS:98118 | %Y% |
| !IN2P3-IPNL | GGUS:98116 | %Y% |
| !IN2P3-IRES | GGUS:98114 | %Y% |
| !IN2P3-LAPP | GGUS:99359 | %Y% |
| !IN2P3-LPSC | GGUS:98117 | %Y% |
| !M3PEC | GGUS:98112 | %Y% |
| MSFG-OPEN | GGUS:98113 | %Y% |
| OBSPM | GGUS:98111 |%Y% |
| RWTH-Aachen | GGUS:98119 | %Y% |
| UNIV-LILLE | GGUS:98115 | %Y% |

---+++ Site Monitoring

25.11.2013: Follow up of the status of sites with Errors raised by glue-validator validating against the GLUE2 profile.

| SITE | GGUS ticket | comments | 
| Australia-ATLAS | GGUS:99115 | %Y% Upgraded - E002: +4000 obsolete entries (app env objects) |
| BEIJING-LCG2 | GGUS:99116 | %Y% Upgraded - E002: 4 obsolete entries (WMS)  |
| BEgrid-ULB-VUB | GGUS:99117 | %X% Not upgraded but obsolete entries gone - E002: 6 obsolete entries (CREAM) |
| Belgrid-UCL | GGUS:98995 <br> GGUS:98996 | %Y% Upgraded - E002: 19 obsolete entries (app env objects) <br> %Y% Condor queue. GLUE1 OK but GLUE2 incorrect. Sys admin wrote info provider for GLUE 2 and now correct. E022, E023 and E024: Default values are published. 3 shares affected in the same CREAM CE|
| CA-ALBERTA-WESTGRID-T2 | GGUS:98987 <br> GGUS:98989 | %Y% Upgraded - E002: 901 obsolete entries (app env objects) <br> E022, E023 and E024: Default values are published. 4 shares affected in the same CREAM CE |
| CA-MCGILL-CLUMEQ-T2 | GGUS:99118 | %Y% Reconfigured CEs - E022, E023 and E024: Default values are published. 5 shares affected in the same CREAM CE |
| CA-SCINET-T2 | GGUS:99119 <br> GGUS:98997 | E002: +1000 obsolete entries (app env objects) <br> E022, E023 and E024: Default values are published. 3 shares affected in the same CREAM CE |
| CA-VICTORIA-WESTGRID-T2 | GGUS:99121 | %Y% Upgraded - E002: +2000 obsolete entries (app env objects) | 
| CSCS-LCG2 | GGUS:99129 | %X% Due to ARC validity problem - E002: 38 obsolete entries |
| CYFRONET-LCG2 | GGUS:99135 | %Y% VOBOX no longer runs resource BDII. E002: 4 obsolete entries (several services) |
| FZK-LCG2 | GGUS:99136 | %Y% Upgraded - E002: 4 obsolete entries |
| GRIF | GGUS:98990 | E002: 168 obsolete entries (objects from several CREAM CEs and RTEs) |
| IEPSAS-Kosice | GGUS:99137 | %Y% Upgraded - E002: +1000 obsolete entries (app env objects) |
| IFIC-LCG2 | GGUS:99138/GGUS:98998 <br> GGUS:99139/GGUS:98999 | %Y% Upgraded - E002: 33 obsolete entries <br> %Y% /var/tmp/info-dynamic-scheduler-generic deleted - E022, E023 and E024: Default values are published. 32 shares affected |
| IL-TAU-HEP | GGUS:99140 | %Y% var/tmp/info-dynamic-scheduler-generic was missing. E022, E023 and E024: Default values are published. 12 shares affected |
| !IN2P3-IRES | GGUS:99141 | E002: 10 obsolete entries (several services) |
| !IN2P3-LPC | GGUS:99142 | %Y% Upgraded - E002: +1000 obsolete entries (app env objects) |
| INFN-BARI | GGUS:99143 | %Y% Upgraded and decommissioned old CEs - E002: 11 obsolete entries (app env objects) |
| INFN-CATANIA | GGUS:99144 | %Y% Upgraded - E002: 6 obsolete entries (several services) |
| INFN-CNAF-LHCB | GGUS:99145 <br> GGUS:99146 | %Y% Upgraded - E002: +1000 obsolete entries (app env objects) <br> %Y% CE upgraded - E022, E023 and E024: Default values are published. 6 shares affected |
| INFN-FRASCATI | GGUS:99147 | %Y% Upgraded - E002: +2000 obsolete entries (app env objects) |
| INFN-MILANO-ATLASC | GGUS:99149 <br> GGUS:99150 | %Y% Upgraded - E002: +300 obsolete entries (app env objects) <br> %Y% A lot of manual config for Condor. Some files needed by info providers were missing - E023 and E024: Default values are published. 10 shares affected |
| INFN-NAPOLI-ATLAS | GGUS:99151 <br> GGUS:99152 | %Y% Upgraded - E002: +1000 obsolete entries (app env objects) <br> %Y% "ldap" user missing from maui - E022, E023 and E024: Default values are published. 12 shares affected |
| INFN-PISA | GGUS:99153 | %Y% CE upgraded - E022, E023 and E024: Default values are published. 6 shares affected |
| KR-KISTI-GCRT-01 | GGUS:99154 | %Y% Upgraded - E002: 3 obsolete entries (site !BDII) |
| KR-KISTI-GSDC-01 | GGUS:99155 | %Y% Upgraded - E002: 3 obsolete entries (site !BDII) |
| NCP-LCG2 | GGUS:99156 | E002: 6 obsolete entries (several services) |
| !NDGF-T1 | GGUS:99157 | %X% Validity is very short for many objects (60s) |
| NIHAM | GGUS:99158 | %Y% Missing lrms_backend_cmd in /etc/lrms/scheduler.conf because in yaim command not all the config targets were specified at once. E023 and E024: Default values are published. 3 shares affected |
| NIKHEF-ELPROD | GGUS:99159 | %Y% Upgraded - E002: +500 obsolete entries (app env objects) |
| PSNC | GGUS:99160 | %Y% Upgraded - E002: 36 obsolete entries (several services) |
| RAL-LCG2 | GGUS:99161 <br> GGUS:99162 | %Y% Upgraded - E002: 15 obsolete entries (several services) <br> %Y% Condor, ARC and CREAM, lots of manual configuration and tuning. E022, E023 and E024: Default values are published. 30-70 shares affected |
| RO-07-NIPNE | GGUS:99163 | %Y% Upgraded - E002: +200 obsolete entries (app env objects) |
| RRC-KI | GGUS:99164 | %Y% Upgraded - E002: +1000 obsolete entries (app env objects) |
| SFU-LCG2 | GGUS:99165 | %Y% emi-torque-utils had to be reinstalled. E022, E023 and E024: Default values are published. 4 shares affected |
| !SiGNET | GGUS:99166 <br> GGUS:99167 | %Y% Upgraded - E002: 5 obsolete entries (app env objects) <br> %Y% Obsolete CE - E022, E023 and E024: Default values are published. 2 shares affected |
| TECHNION-HEP | GGUS:99168 | %Y% var/tmp/info-dynamic-scheduler-generic was missing. E022, E023 and E024: Default values are published. 12 shares affected |
| TR-03-METU | GGUS:99169 | %Y% Upgraded - E002: 6 obsolete entries (app env objects) |
| TR-10-ULAKBIM | GGUS:99170 | %Y% Upgraded - E002: 11 obsolete entries (app env objects) |
| UB-LCG2 | GGUS:99171 | %Y% Upgraded - E002: 2 obsolete entries (app env objects) |
| UKI-LT2-IC-HEP | GGUS:99172 | %Y% Upgraded - E002: 3 obsolete entries (app env objects) |
| UKI-LT2-RHUL | GGUS:99173 | %Y% Upgraded - E002: +2000 obsolete entries (app env objects) |
| UKI-LT2-UCL-HEP | GGUS:99174 <br> GGUS:99176 | %Y% Upgraded - E002: +900 obsolete entries (app env objects) <br> %Y% Upgraded to SL6 and fixed. E022, E023 and E024: Default values are published. 3 shares affected |
| UKI-NORTHGRID-LANCS-HEP | GGUS:99177 | %Y% Upgraded - E002: 11 obsolete entries (app env objects) |
| UKI-NORTHGRID-LIV-HEP | GGUS:99178 | %Y% Upgraded - E002: +4000 obsolete entries (app env objects) |
| UKI-NORTHGRID-MAN-HEP | GGUS:98994 | %Y% Upgraded - E002: obsolete entries (app env objects) |
| UKI-SCOTGRID-ECDF | GGUS:99179 <br> GGUS:99180 | %X% Old service to be decommissioned E002: +2000 obsolete entries (app env objects) <br> E022, E023 and E024: Default values are published. 13 shares affected |
| UKI-SOUTHGRID-RALPP | GGUS:100480 | %Y% Upgraded - E002: +1000 obsolete entries (app env objects) |
| UNIBE-LHEP | GGUS:99182 | %Y% Upgraded - E002: +2000 obsolete entries (app env objects) |
| ifae | GGUS:99183 | %Y% Fixed site configuration. E022, E023 and E024: Default values are published. 18 shares affected |
| pic | GGUS:99184 | %Y% Problem when batch system is also used by non grid users that are unknown to the CE and crashes. E022, E023 and E024: Default values are published. 210 shares affected |
| praguelcg2 | GGUS:99185 | %Y% Upgraded - E002: +1000 obsolete entries (app env objects) |
| ru-Moscow-FIAN-LCG2 | GGUS:99187 | %Y% Upgraded - E002: 3 obsolete entries (several services) |
| ru-PNPI | GGUS:99188 | %Y% Upgraded - E002: 6 obsolete entries (several services) |

---+++ Storage Share IDs

The following table tracks GGUS tickets opened to LHCb Tier 1 sites who publish many Storage Shares. Sometimes it is difficult to understand what type of storage has been allocated in the share and why so many shares need to be defined.

| Site | GGUS tickets | comments |
| FZK-LCG2 | GGUS:99750 | Storage Share names come from dCache configuration. Explanations from Paul Millar | 
| SARA-MATRIX | GGUS:99809 | %Y% Confirmed share names and updated them in the dashboard script |
| !IN2P3-CC | GGUS:99875 | %Y% Confirmed share names and updated them in the dashboard script |
| INFN-T1 | GGUS:99888 | %Y% Confirmed share names and updated them in the dashboard script  |
| RAL-LCG2 | GGUS:99889 | %Y% Confirmed share names and updated them in the dashboard script |

---+++ Missing Mapping Policy objects

The following table tracks GGUS tickets opened to sites who do not publish the mapping policy for computing shares. This doesn't allow to query computing shares allocated for a particular VO, which is something needed to monitor the Max CPU time attribute for the LHCb VO.

| Site | GGUS tickets | comments |
| !IN2P3-CPPM | GGUS:100223 | Problem with the quattor configuration |
| GRIF | GGUS:100222 | Same as above |
| !IN2P3-LAPP | GGUS:100221 | Same as above |
| !IN2P3-CC-T2 | | |

---+++ Cleaning SW Tags

The following table tracks GGUS tickets opened to sites whose CEs are preventing VO managers to clean SW tags for their VO.

| Site | GGUS tickets | comments |
| LHCb |^|^|
| pic |  GGUS:101037 | %Y% SW tags published in a CE not allocated for LHCb. They are deleted now | 
| UKI-SCOTGRID-ECDF | GGUS:101038 | |
| BG03-NGCC | GGUS:101039 | %Y% No LHCb tags defined in the CE according to the sys admin since the LHCb VO is not supported at the site  |
| ATLAS |^|^|
| CA-MCGILL-CLUMEQ-T2 | GGUS:101041 | %Y% SW tags coming from a test CE have been deleted by the site |
| CYFRONET-LCG2 | GGUS:101042 | %Y% tags seem to be deleted now by the site |
| RO-07-NIPNE | GGUS:101043<br>GGUS:106312 | %Y% atlas tags are published and CE is accessible for ATLAS. Alessandro could try again to delete the tags<br> %Y% Not sure where tags are published as they are not published by their CE (unless it's tbit01.nipne.ro, but it's refusing Alessandro's attempts). Sys admin deleted the tags. |
| UKI-LT2-QMUL | GGUS:101044 | %Y% site reconfigured the common tags area. Alessandro checked that there were no tags |
| INFN-ROMA2 | GGUS:101049 | %Y% Site was in downtime for a long time. Tags could be deleted now |
| TUDresden-ZIH | GGUS:101150 | %Y% SW tags could be eventually deleted |
| BG03-NGCC | GGUS:106313 | %Y% Alessandro has just tried and cannot remove the tags from their CE (ce02.ngcc.acad.bg). The sys admin has deleted the tags |
| ITEP | GGUS:106314| %Y% Cannot remove the tags, get "system error in unlink" even if using uberftp to remove the tag file. Sys admin has deleted the tags |
| UKI-NORTHGRID-LANCS-HEP | GGUS:106316 | %Y% Cannot remove the tags, get "system error in unlink" even if using uberftp to remove the tag file. Sys admin confirmed permissions of the tag area were 'root' instead of 'sgmatlas'. He has now deleted the SW tags |
|CMS|^|^|
| !BelGrid-UCL | GGUS:106813 | %X% |
| INDIACMS-TIFR | GGUS:106814 | %Y% Correct role given to Christoph and tags deleted |
| INFN-PADOVA | GGUS:106815 | %Y% Christoph deleted the tags after the site made sure he could do it with the correct role |
| Kharkov-KIPT-LCG2 | GGUS:106816 | %Y% Tags deleted by the site |
| Ru-Troitsk-INR-LCG2 | GGUS:106817 | %Y% Tags probably deleted by the site |
| TW-NCUHEP | GGUS:106818 | %X% |
| UKI-LT2-RHUL | GGUS:106819 | %Y% Tags deleted by the site |
| UKI-NORTHGRID-SHEF-HEP | GGUS:106820 | %Y% Tags deleted by the sys admin |

---+++ T1 Storage Deployment

This table tracks GGUS tickets opened to T1s to be able to publish in the !BDII coherent storage types and versions and supported VOs as tracked in the [[http://wlcg-mon.cern.ch/dashboard/request.py/siteview#currentView=T1+Storage+Deployment][Dashboard]].

| Site | GGUS tickets | comments |
| CERN | None, contact by mail | %X% Requested to have a more compact EOS versioning syntax and to remove the "unknown" string from the Castor release |
| RAL | GGUS:106480 | %Y% Requested to publish meaningful version. Fixed the versioning in the dashboard scripts that were not able to parse correctly ":" |
| BNL-ATLAS | GGUS:106483 | %Y% Requested to publish storage version. BNL answered that the Classic SE should not be taken into account. Removed from table. |
| USCMS-FNAL-WC1 | GGUS:106504 | %Y% Requested to publish storage version. FNAL confirms Classic SE doesn't need to be taken into account |
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who
pdf	WLCG-GLUE2-validation-statistics-July2013.pdf	r1	manage	513.5 K	2013-08-09 - 12:52	MariaALANDESPRADILLO
pdf	WLCG-GLUE2-validation-statistics-June2013.pdf	r1	manage	941.3 K	2013-06-21 - 11:41	MariaALANDESPRADILLO
pdf	WLCG-GLUE2-validation-statistics-May2013.pdf	r1	manage	877.9 K	2013-05-16 - 10:51	MariaALANDESPRADILLO
Topic revision: r108 - 2014-08-13 - MariaALANDESPRADILLO
EGEE
Main.WebList
Welcome Guest
- Cern Search
- TWiki Search
- Google Search
EGEE All webs
Copyright &© by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Ask a support question or Send feedback