AtlasGridInformationSystemAddModifyData
Introduction
This page is for organizing and representation of user requirements, use cases and instructions for the updating of information in AGIS.
Main types of changeable information
AGIS interfaces should provide possibility to add, change next information:
- ATLAS Site definition * Relation with real infrastructure provider (OIM/GOCDB site)
- DDM Endpoint definition * DDM topology information * Quota declaration * Permission declaration
- PANDA site / site id (resource) / queue information
- Service related information:
- Declare new Storage Element of change some information about SE
- Declare and modify Computing Element
- Declare and modify Redirector service
- Declare and modify LFC service (Obsolete)
User requirements
Main users requirements: to have possibility to add/modify ATLAS related information in AGIS by using different interfaces.
There are two main interfaces for this: API (for users application) and Web-based user interface.
Use Cases
Authorization / Authentication
To be able to add/modify information in AGIS through the
WebUI client should be authorized. Authorization is based on X.509 certificate, which should be imported into browser.
Access permissions can be requested from the main
WebUI page (Request ADMIN privileges link:
http://atlas-agis.cern.ch/agis/user/register/
)
ObjectStore declaration
Please check detailed steps to declare ObjectStore services and configure OS DDMEndpoints here:
PanDA/DefineObjectStore
Removing an object
For most of objects in AGIS deletion operation disallowed for user. This restriction came from security reasons. If you want to delete something please, contact AGIS team ( atlas-adc-agis AT cern.ch
AGIS How To Delete ).
Add/Edit ATLAS Site
According to rules of ADC, ATLASSite should be associated with GOCDB/OIM site. So, before starting procedure of creation ATLASSite be sure, that site already registered in GOCDB or OIM. INformation about GOCDB/OIM sites collects by AGIS automatically.
For creating new site through
WebUI click “Add ATLASsite” link on main page of AGIS
WebUI, and if your certificate properly installed, and you have proper permissions ‘ATLASSite creation form’ will be open.
According this form, next values should be defined:
- OIM/GOCDB Site: select from list of sites, known by AGIS
- ATLAS Name: specify unique name
- ATLAS Tier Level: Tier of site (1,2,3)
- Cloud realtion
- DDM Name: specify DDM name
- .... some other option
AGIS
WebUI will not give possibility to save inconsistent information.
Form for editing ATLAS site information similar to the form for creation, just extended with
State and
State comment fields.
Please, check video tutorial (example) to see basic usage of forms:
Add/Edit DDMEndpoint
Changing of DDM related information – is one of most common operation. During historical reasons, some common (known) format was introduced. AGIS forms for add/modify DDM information was created to be similar to this format.
For creating new DDMEndpoint through
WebUI click “Add ATLASsite” link on main page of AGIS
WebUI. In case you have enough permissions, Add DDM endpoint form, will be open.
Next values in this form should be defined (most of them known for ADC experts):
- Name of DDM Endpoint: Endpoint name in format [SITE NAME]_[TOKEN]
- Token: select from list, for correct association with token
- Storage endpoint: srm or gridftp URL
- Domain: regular expression which used for extract domain information from ‘Storage endpoint’
- Is tape:
- ToolAssigner: by default ‘lcg’, not needed to be changed for most of case
- Use Cloud LFC or Served LFC: this fields define LFC service which should serve DDMEndpoint, if ‘Use Cloud LFC’ cheked, LFC service will be defined automatically (OBSOLETE)
- Default FTS: default FTS service for endpoint (additional FTS service can be defined later) Permissions: select ‘Permission group’ – for most of cases ‘default’ can be used
After filling form, intermediate step with some validation will be performed. In case validation successful and results of discovering some additional info satisfactory, will be offered the opportunity to save the data.
Form for changing information extended with some additional functionality: provide possibility change state of endpoint (‘ENABLED’, ‘DISABLED’), define quotas for different groups.
Bulk update of FTS server for DDMEndpoints sharing the same storage endpoint
[voatlas290] /afs/cern.ch/user/d/ddmusr01 > head -5 /var/tmp/.dq2ddmusr01/ToACache.py
...
...
[voatlas290] /afs/cern.ch/user/d/ddmusr01 > rm /var/tmp/.dq2ddmusr01/ToACache.py
...
[voatlas290] /afs/cern.ch/user/d/ddmusr01 > dashb-agent-stop dq2agents
...
[voatlas290] /afs/cern.ch/user/d/ddmusr01 > dashb-agent-start dq2agents
...
[voatlas290] /afs/cern.ch/user/d/ddmusr01 > head -5 /var/tmp/.dq2ddmusr01/ToACache.py
...
--> you're done.
Add/Edit Panda Site
To add new
PandaSite please, make next steps:
- On AGIS WebUI main page click "Define PanDA site"
- Fill form:
- Select related ATLAS Site (start typing and possible variants will be advised)
- Type desired PandaSite name
- Click 'Check input data'
- Some additional information will appear (for example: list of already associated PandaSites for selected ATLAS Site)
- In case inputed data are correct - 'Save' button will appear.
- Click 'Save' button.
New
PandaSite will be created only after clicking of 'Save' button.
After saving, you will be redirected to page with description of the
PandaSite. You will have able to create Panda Resourse (
PandaSiteID) directly from this page.
Add/Update/Delete PanDA resources (also known as 'PanDA Site ID')
- Add
- There are two ways to add PanDA resource:
- from main page click "Define PANDA resource" link
- from description of PanDA site page click "Add PANDA resource for this PANDA site..."
- only five parameters should be defined and only two of them obligatory:
- PANDA Site (obligatory parameter, start typing and possible variants will be advised)
- Name of PANDA resource (obligatory parameter)
- CVMFS
- Default analysis
- Default production
- After defining the parameters you should click "Check input data", some additional info will appear. If the input data are acceptable "Save PANDA resource" button will appear. Data will be saved only after clicking of "Save PANDA resource" button.
- Update
- To edit PanDA resource description you should click "Update PANDA resource info" link on PanDA resource information page.
- Update PanDA resource forms has similar views as the Add one (above), only five parameters should be defined and only two of them obligatory
- After defining of needful parameters you should click "Check input data", some additional info will appear. If inputted data acceptable "Save PANDA resource" button will appear. Data will be saved only after clicking of "Save PANDA resource" button.
- Delete
- click the button "Delete PANDA Resource" in the "update" page (see above)
Add/Clone/Edit/Delete PANDA Queue
Modification of
PanDA Queue information is very common and often operation. To add new
PanDA Queue you should click "Define PANDA queue" link on AGIS
WebUI main page.
After this form of definition of
PanDA queue will be open, only couple of parameters in "PanDA queue" definition in AGIS are mandatory, but total number of parameters are huge.
To provide some advice with filling form, some features was implemented: "example", "clone". If you already know, that new
PanDA queue will be similar with one which already existing you can type name of this queue in 'Specify PANDA Queue' field (autocomplete will give some variants, just start typing). After this, if you will click on "Show as example" button - description of selected
PanDA queue will fill right side of form. If you will click 'Clone' button - you will be redirected to clone form: left side of form will be filled with values of selected queue and same values will be shown on right side as example.
There are no additional validation step for creation of
PanDA queue, but nothing will be saved if data incomplete or wrong. After successful saving of
PanDA queue description you will be redirected to "PanDA Queue edit" form. On this form you will be able to modify parameters or manage
PanDA queue - CE queue relation
- remove a panda queue
- To delete PanDA queue you should:
- find PanDA queue in PanDA queues list (Section 'PANDA queues' + filtering)
- click PanDA queue name (left column)
- click 'Update PANDAQueue and CE associated info' link
- click 'Delete PANDA queue' button (bottom right side of page)
- confirm deletion (Alert box will appear)
- removal of a panda queue will be propagated to
- panda scheadconfig
- pilot factory config
- Parameters that cannot be changed in AGIS
-
comment, multicloud, nqueue, space, tspace, status, status_control, status_override,
Rules for adding DDM endpoints to PanDA queues as of January 2018
PanDA server components started using the AGIS "astorages" field end of 2017 to increase the flexibility associating DDM endpoints to
PanDA queues. An overview of the possibilities is given in the image below.
- You can attach ddm endpoints for input/read ("read_lan") belonging to different sites. The endpoint with "read_lan0" will get the potential input file transfers for the jobs.
- The pilot will use one of the attached input endpoints (it will not use other endpoints!) and it is the Rucio list replica function that orders them by closeness.
- Output endpoints have to belong to the same site and there can only be one endpoint of each spacetoken type (e.g. DATADISK, SCRATCHDISK, etc.). The endpoint with "write_lan0" will be the default endpoint for writing the output.
- Input and output endpoint do not have to belong to the same site.
Associate PandaQueue with CE Queue
Before AGIS the information on which CE (gatekeeper+localqueue) should have been used for a pandaqueue was stored in the "gatekeeper" and in "localqueue" attribute. Those attributes are still in the schedconfig visualization, but they are
not used anymore.
Wit AGIS now you may associate multiple CEs to the same pandaqueue. We recommend you to make just one pandaqueue for each
PandaResource(
PandaSiteID), and associate to this pandaqueue many CEs.
To associate
PanDA queue and CE queue you should make next steps:
- Find or define PanDA queue. To find PanDA queue you should navigate to 'PANDA Queue' section of WebUI. (e.g. from http://atlas-agis.cern.ch/agis/atlassite/main/Australia-ATLAS/
click on the left menu on Panda (within ATLAS Info)) You will see the list of PanDA queues which can be sorted by different parameters (PanDA queue name, PanDA Site, ATLAS Site etc.)
- Navigate to PanDA queue description page: click to PanDA queue name in table (e.g. http://atlas-agis.cern.ch/agis/panda_queue/detail/ANALY_AUSTRALIA/full/
)
- Navigate to PanDA queue editing page: click "Update PANDAQueue and CE associated info" link
- In bottom of page you will find "Associated CE/Queue" section. This section will represent information about already associated CE queues with possibility of manage relations.
- To add new association you should click 'Find and associate another CE/Queue' click
- Search form will appear. This form needed to restrict list of possible CE queues. Please, enter name (or part of name) of CE, site or queue and click 'Search' button
- A list of queues satisfying will appear
- To create the link between PanDA queue and CE queue, please check CE queue and click 'Save'
Add/Edit Computing Element service and CE queue
In most of cases information about services (CE, SE, etc.) collected by AGIS automatically from GOCDB, OIM and other sources, but in exceptional cases, such information can be added using the
WebUI.
To add new CE into AGIS you should:
- Click 'Define CE service' link on main page of AGIS WebUI
- Fill form 'Computing Element Creation'
- Field 'OIM/GOCDB site' extended with autocomplete - start typing and variants will advised.
- 'AGIS CE name' - unique name of service in AGIS, usually looks like [Site name]-CE-[endpoint]
- 'Endpoint' - hostname and port [hostname]:[port]
- 'Job manger', 'Flavour', 'Version' - parameters, which describe job management software of CE
- Click "Check input data" button, in case data acceptable "Save & continue" button appear.
- Click "Save & continue" - only after this data will be saved.
After creation of CE you will be redirected to "Computing Element Editing" form, which is similar to creation, but have two extra fields - 'State' and 'State comment'.
To edit CE service information you should:
- find service: 'Service' section of WebUI + some filtering will help you
- Click Service name: 'Computing element description' page will appear
- Click 'Detailed info' (top right)
- Click 'Edit CE description': "Computing Element Editing" form will appear
To add CE queue to Computing element you should:
- find service: 'Service' section of WebUI + some filtering will help you
- Click Service name: 'Computing element description' page will appear
- Click 'Detailed info' (top right)
- Click 'Add Queue' link: "Queue Creation" form will appear
"CE" field of form will be pre filled with correct value. You should fill only 'Queue name' value.
Add/Edit LFC service (OBSOLETE)
To create new LFC service click "Define LFC service" link on main page of AGIS
WebUI.
According to the appeared form, next values should be filled:
- OIM/GOCDB Site: select from list of sites, known by AGIS
- ATLAS Service Name: specify some short name of service which will be used to present the service in AGIS WebUI
- Endpoint: hostname of machine with lfc:// prefix and valid path :/grid/atlas postfix
WebUI will not allow to add inconsistent information to AGIS.
Form for editing information is similar to the form for creation, and have two additional fields: State and State comment.
Add/Edit Storage Element service
To create new Storage Element service click "Define SE service" link on main page of AGIS
WebUI.
According to the appeared form, next values should be filled:
- OIM/GOCDB Site: select from list of sites, known by AGIS
- Endpoint: hostname of machine
- SE type: type of storage element should be selected
- Redirector: if SE type XROOTD or HTTP is selected, it will be possible to define Redirector for the service
WebUI will not allow to add inconsistent information to AGIS. If information is correct, AGIS unique service name will be generated automatically and will appear on the second page.
Form for editing information is similar to the form for creation, and have two additional fields: State and State comment.
Add/Edit Redirector service
To create new Redirector service click "Define Redirector" link on main page of AGIS
WebUI.
According to the appeared form, next values should be filled:
- OIM/GOCDB Site: select from list of sites, known by AGIS
- Endpoint: hostname of machine with prefix and port
- Parent redirector: if redirector is being added has parent redirector, it should be selected from the dropdown list, if not, leave it blank
- Protocol
WebUI will not allow to add inconsistent information to AGIS. If information is correct, AGIS unique service name will be generated automatically and will appear on the second page.
Form for editing information is similar to the form for creation, and have two additional fields: State and State comment.
REST API to (bulk) update PandaQueue object properties
To bulk update the
PandaQueue attributes one should POST JSON encoded input (for BULK operation) or apply GET request (for simple update) to the following URI:
https://atlas-agis-api.cern.ch/request/pandaqueue/update/set_attributes/?json
json_pretty=True flag can be passed to show results in human readable JSON format.
An example of BULK update using curl:
curl --capath /etc/grid-security/certificates --cacert $X509_USER_PROXY --cert $X509_USER_PROXY -X POST -d @pq_setattributes.json "https://atlas-agis-api.cern.ch/request/pandaqueue/update/set_attributes/?json&json_pretty=True" > pq_setattributes_out.json
where input
pq_setattributes.json file contains the list of dict entries with defined
pandaqueue object attributes to be updated
e.g.
[
{"name": "AGLT2_MCORE-condor", "capability":"mcore", "corecount":8, "direct_access_wan":true, "description":"new desc value here"},
{"name": "BAD_NAME", "capability":"mcore"}
]
Server responses with the list of entries like this:
[ PandaQueue name, return_val, "error message in case of failure (when return_val is None)" ]
in case of real DB update the return_val is True, otherwise False means no DB modifications
in out example the response was:
cat pq_setattributes_out.json
[
[
"AGLT2_MCORE-condor",
false,
""
],
[
"BAD_NAME",
null,
"(agis.utils.exceptions.NonExistsException) Failed to resolve PandaQueueObject by query: {'name': u'BAD_NAME'}, not found!"
]
]
Required input values to be passed as a dict:
['name']
REST API Summary:
- API format:
- Real example:
https://atlas-agis-api.cern.ch/request/pandaqueue/update/set_attributes/?json&json_pretty=1&name=ANALY_BNL_MCORE-condor&capability=mcore&description=multicore+analysis+queue
-
curl --capath /etc/grid-security/certificates --cacert $X509_USER_PROXY --cert $X509_USER_PROXY -X POST -d @data.json
"https://atlas-agis-api.cern.ch/request/pandaqueue/update/set_attributes/?json" > results.json
In case of wrong attribute passed, full list of accepted parameters will be exposed in error message.
By default, if *_on flag (indicating if property value should be inherited from parent PQ object or not) is not passed to the input, it will be triggered to ON once corresponding PQ.attribute is updated.
PQ data can be fetched and checked from the
PandaQueue schedconfig JSON export:
http://atlas-agis-api.cern.ch/request/pandaqueue/query/list/?json&preset=schedconf.all
REST API to (bulk) update DDMEndpoint protocols
To bulk update the protocols data one should POST JSON encoded input to the following URI:
https://atlas-agis-api.cern.ch/request/ddmendpoint/update/set_protocols/?json
an example using curl:
curl --capath /etc/grid-security/certificates --cacert $X509_USER_PROXY --cert $X509_USER_PROXY -X POST -d @protocols.json "https://atlas-agis-api.cern.ch/request/ddmendpoint/update/set_protocols/?json" > set_protocols_out.json
where input
protocols.json file contains the list of dict entries with defined
ddmendpoint, se, activity, priority, endpoint, is_default keys
e.g.
[
{"ddmendpoint": "AGLT2_LOCALGROUPDISK", "se":"https://head01.aglt2.org:2880", "activity":"rwd", "priority":1, "endpoint":"/atlaslocalgroupdisk/rucio/"},
{"ddmendpoint": "AGLT2_DATADISK_BAD", "se":"srm://head01.aglt2.org:8443/srm/managerv2?SFN=", "activity":"s", "priority":1, "endpoint":"/test/hhh"}
]
The activity parameter takes either single value ("r") or the list of activities ("rwd")
Server responses with the list of entries like this:
[ (DDM, SE, activity), return_val, "error message in case of failure (when return_val is None)" ]
in case of real DB update the return_val is True, otherwise False means no DB modifications
in out example the response was:
cat set_protocols_out.json
[
[
[
"AGLT2_LOCALGROUPDISK",
"https://head01.aglt2.org:2880",
"r"
],
true,
""
],
[
[
"AGLT2_LOCALGROUPDISK",
"https://head01.aglt2.org:2880",
"w"
],
true,
""
],
[
[
"AGLT2_LOCALGROUPDISK",
"https://head01.aglt2.org:2880",
"d"
],
true,
""
],
[
[
"AGLT2_LOCALGROUPDISK",
"https://head01.aglt2.org:2880",
"d"
],
null,
"(agis.utils.exceptions.NonExistsException) Failed to resolve DDMEndpoint by name=AGLT2_DATADISK_BAD, not found!"
]
the primary key of data entry is {ddm, se, activity}
The deletion of DDMEndpoint protocols can be done in the same way
by posting JSON encoded data to the URI:
https://atlas-agis-api.cern.ch/request/ddmendpoint/update/remove_protocols/?json
Required input values to be passed as a dict:
['ddmendpoint', 'se', 'activity']
REST API Summary:
- API format:
- Real example:
https://atlas-agis-api.cern.ch/request/ddmendpoint/update/set_protocols/?
json&ddmendpoint=AGLT2_DATADISK&se=srm://head01.aglt2.org:8443/srm/managerv2?
SFN=&activity=rwd&priority=3&endpoint=/pnfs/aglt2.org/atlasdatadisk/rucio/
* curl --cert $X509_USER_PROXY --key $X509_USER_PROXY -kS -X POST -d @protocols.json
"https://atlas-agis-api.cern.ch/request/ddmendpoint/update/set_protocols/?json" > set_protocols.json
Protocols data can be fetched from DDMEndpoint JSON export:
http://atlas-agis-api.cern.ch/request/ddmendpoint/query/list/?json
REST API to (bulk) update PandaQueue status (PanDA blacklisting)
AGIS provides REST API to (bulk) update the blacklisitng information.
it's possible to do it either via sending HTTP POST requests (more convenient for bulk operations) as well as sending HTTP GET request
To bulk update pandaqueue blacklisting statuses one should POST JSON encoded input to the following URI:
https://atlas-agis-api.cern.ch/request/pandaqueuestatus/update/set_probestatus/?json
an example using curl:
curl --cert $X509_USER_PROXY --key $X509_USER_PROXY -kS -X POST -d @blacklisting.json "https://atlas-agis-api.cern.ch/request/pandaqueuestatus/update/set_probestatus/?json" > output.json
where input
blacklisting.json file contains the list of dict entries with defined
pandaqueue, probe, value, activity, reason, expiration keys
e.g.
[
{"pandaqueue": "AGLT2_TEST-condor", "activity":"a", "value":"BROKEROFF", "reason":"HC changed status", "probe":"hummercloud", "expiration":"2016-06-07T12:00:00"},
{"pandaqueue": "ANALY_MCGILL", "value":"TEST", "reason":"HammerCloud PFT test suite", "probe":"hummercloud", "expiration":"2016-06-08T12:00:00"},
]
The activity parameter can be omitted.
Server responses with the list of entries like this:
[ (PandaQueue, activity), return_val, "error message in case of failure (when return_val is None)" ]
in case of real DB update the return_val is True, otherwise False means no DB modifications
Single update operation can be send using HTTP GET query:
https://atlas-agis-api.cern.ch/request/pandaqueuestatus/update/set_probestatus/?json&pandaqueue=AGLT2_TEST-condor&value=BROKEROFF&reason=TEST&expiration=2016-06-07T12:00:00&probe=hammercloud&activity=a
REST API Summary:
- Real example:
- Single update
curl --cert $X509_USER_PROXY --key $X509_USER_PROXY "https://atlas-agis-api.cern.ch/request/pandaqueuestatus/update/set_probestatus/?json&pandaqueue=AGLT2_TEST-condor&value=BROKEROFF&reason=TEST&expiration=2016-06-07T12:00:00&probe=hammercloud&activity=a"
- Bulk update:
curl --cert $X509_USER_PROXY --key $X509_USER_PROXY -kS -X POST -d @blacklisting.json "https://atlas-agis-api.cern.ch/request/pandaqueuestatus/update/set_probestatus/?json" > output.json
Blacklisting details can be fetched from JSON export:
http://atlas-agis-api.cern.ch/request/pandaqueuestatus/query/list/?json
History of Blacklisting details can be fetched from JSON export:
http://atlas-agis-api.cern.ch/request/pandaqueuestatus/historty/list/?json
List of accepted keys and possible filters can be resolved from the API doc page:
http://atlas-agis-api.cern.ch/request/pandaqueuestatus/query/help/
OLD after the 28th February 2013 -- Temporary instructions during the period in which both AGIS and svn SchedConfig python file are used.
Major updates:
--
AleDiGGi - 30-Mar-2012
--
DanilaOleynik - 11-Dec-2012
--
DanilaOleynik - 13-Jan-2013
--
DanilaOleynik - 16-Jan-2013
Responsible:
AleDiGGi
Last reviewed by:
Never reviewed