AtlasGridInformationSystemAddModifyData < AtlasComputing

AtlasComputing Web>AtlasComputing>AtlasDistributedComputing>AtlasGridInformationSystem>AtlasGridInformationSystemAddModifyData (2019-11-28, AlexeyAnisyonkov) (raw view)
EditAttachPDF
<!--
   * Set ALLOWTOPICVIEW = Main.AllUsersGroup
-->

%CERTIFY%
---+!!<nop>%TOPIC%
%TOC%     <!-- this line is optional -->
%STARTINCLUDE%

---+Introduction
This page is for organizing and representation of user requirements, use cases and instructions for the updating of information in AGIS.

---++ Main types of changeable information
AGIS interfaces should provide possibility to add, change next information:

   *    ATLAS Site definition
       *    Relation with real infrastructure provider (OIM/GOCDB site)
   *   DDM Endpoint definition
       *    DDM topology information
       *    Quota declaration
       *    Permission declaration
   *   PANDA site /  site id (resource) / queue information
   *   Service related information:
      *   Declare new Storage Element of change some information about SE
      *   Declare and modify Computing Element
      *   Declare and modify Redirector service 
      *   Declare and modify LFC service (Obsolete) 
 
---+User requirements

Main users requirements: to have possibility to add/modify ATLAS related information in AGIS by using different interfaces. 
There are two main interfaces for this: API (for users application) and Web-based user interface.

---+Use Cases
<!-- Add the main topic here. i.e. create some new headings as follows: -->

---++ Authorization / Authentication 

To be able to add/modify information in AGIS through the WebUI client should be authorized. Authorization is based on X.509 certificate, which should be imported into browser. 
Access permissions can be requested from the main WebUI page (Request ADMIN privileges link: http://atlas-agis.cern.ch/agis/user/register/)


---++ !ObjectStore declaration

Please check detailed steps to declare !ObjectStore services and configure OS DDMEndpoints here: [[PanDA/DefineObjectStore]]


---++ Removing an object

For most of objects in AGIS deletion operation disallowed for user. This restriction came from security reasons. If you want to delete something please, contact AGIS team (   atlas-adc-agis AT cern.ch [[mailto:?subject=AGIS - Please Delete objects][AGIS How To Delete]] ).
   * PanDA queue objects can be deleted (see [[#Add_Clone_Edit_Delete_PANDA_Queu]])
   * DDM endpoints can be disabled (see [[#Add_Edit_DDMEndpoint]])

---++ Add/Edit ATLAS Site

According to rules of ADC, ATLASSite should be associated with GOCDB/OIM site. So, before starting procedure of creation ATLASSite be sure, that site already registered in GOCDB or OIM.  INformation about GOCDB/OIM sites collects by AGIS automatically. 

For creating new site through WebUI click  “Add ATLASsite” link on main page of AGIS WebUI, and if your certificate properly installed, and you have proper permissions ‘ATLASSite creation form’ will be open. 

According this form, next values should be defined:

   *   *OIM/GOCDB Site*: select from list of sites, known by AGIS
   *   *ATLAS Name*: specify unique name
   *   *ATLAS Tier Level*:   Tier of site (1,2,3)  
   *   *Cloud* realtion
   *   *DDM Name*: specify DDM name
   *    .... some other option 

AGIS WebUI will not give 	possibility to save inconsistent information. 

Form for editing ATLAS site information similar to the form for creation, just extended with *State* and  *State comment* fields.

Please, check video tutorial (example) to see basic usage of forms:

   * [[%ATTACHURL%/ATLASSite_create_2.mov][ATLASSite_create_2.mov]]: ATLAS Site create example.
   * [[%ATTACHURL%/ATLASSite_edit.mov][ATLASSite_edit.mov]]: ATLAS Site edit example.
  

---++ Add/Edit DDMEndpoint

Changing of DDM related information – is one of most common operation. During historical reasons, some common (known) format was introduced. AGIS forms for add/modify DDM information was created to be similar to this format.

For creating new DDMEndpoint through WebUI click  “Add ATLASsite” link on main page of AGIS WebUI. In case you have enough permissions, Add DDM endpoint form, will be open.

Next values in this form should be defined (most of them known for ADC experts):

   *   *Name of DDM Endpoint*:  Endpoint name in format [SITE NAME]_[TOKEN] 		  
   *   *Token*: select from list, for correct association with token
   *   *Storage endpoint*: srm or gridftp URL		
   *   *Domain*: regular expression which used for extract domain information from ‘Storage endpoint’   
   *   *Is tape*: 		
   *   *ToolAssigner*: by default ‘lcg’, not needed to be changed for most of case 		
   *   *Use Cloud LFC or Served LFC*: this fields define LFC service which should serve DDMEndpoint, if ‘Use Cloud LFC’  cheked, LFC service will be defined automatically  (OBSOLETE)
   *   *Default FTS*:  default FTS service for endpoint (additional FTS service can be defined later) Permissions: select ‘Permission group’ – for most of cases ‘default’ can be used
	
After filling form, intermediate step with some validation will be performed. In case validation successful and results of discovering some additional info satisfactory, will be offered the opportunity to save the data.
 
Form for changing information extended with some additional functionality: provide possibility change state of endpoint (‘ENABLED’, ‘DISABLED’), define quotas for different groups. 

   * [[%ATTACHURL%/update_ddmendpoint.mov][update_ddmendpoint.mov]]: Update DDM Endpoint example

---+++ Bulk update of FTS server for DDMEndpoints sharing the same storage endpoint
   * start from http://atlas-agis.cern.ch/agis/ddm_endpoint/table_view/ and filter to one of the DDMEndpoints in which you're interested (let's say IN2P3-LPC). Click on one of the DDMEndpoint.
   * you'll be redirected here http://atlas-agis.cern.ch/agis/ddm_endpoint/detail/IN2P3-LPC_DATADISK/full/ . Find the "storage element" attribute in the middle of the page, and click on the clicckable name of the storage endpoint.
   * you'll be now in such page http://atlas-agis.cern.ch/agis/storage_element/detail/2397/ . you have "bulk update" button. click it.
   * you're here https://atlas-agis.cern.ch:2821/agis/ddmendpoint/bulkupdate/?se_name=IN2P3-LPC-SRM-clrlcgse01.in2p3.fr . Select the "default FTS" you want, select the checkbox to apply this to all the DDMendpoints. Click save till you have saved.
   * On the DQ2 SiteServices, to speed up the refresh of !ToACache which has 70k seconds of validity (the AGIS cache is updated each 15 minutes so check that ToAcache is updated ([[http://atlas.web.cern.ch/Atlas/GROUPS/DATABASE/project/ddm/releases/TiersOfATLASCache.py]])) :
<verbatim> 
[voatlas290] /afs/cern.ch/user/d/ddmusr01 > head -5 /var/tmp/.dq2ddmusr01/ToACache.py
...
...
[voatlas290] /afs/cern.ch/user/d/ddmusr01 > rm /var/tmp/.dq2ddmusr01/ToACache.py
...
[voatlas290] /afs/cern.ch/user/d/ddmusr01 > dashb-agent-stop dq2agents
...
[voatlas290] /afs/cern.ch/user/d/ddmusr01 > dashb-agent-start dq2agents
...
[voatlas290] /afs/cern.ch/user/d/ddmusr01 > head -5 /var/tmp/.dq2ddmusr01/ToACache.py
...
--> you're done.
</verbatim>

---++ Add/Edit Panda Site

To add new PandaSite please, make next steps:
  
   * On  AGIS WebUI main page click "Define PanDA site" 
   * Fill form: 
      * Select related ATLAS Site (start typing and possible variants will be advised)
      * Type desired  PandaSite name 
   * Click 'Check input data'
   * Some additional information will appear (for example: list of already associated PandaSites  for selected ATLAS Site)
   * In case inputed data are correct - 'Save' button will appear. 
   * Click 'Save' button.

New PandaSite will be created only after clicking of 'Save' button.

After saving, you will be redirected to page with description of  the PandaSite. You will have able to create Panda Resourse (PandaSiteID) directly from this page. 

---++ Add/Update/Delete !PanDA resources (also known as 'PanDA Site ID')

   $ *Add* : There are two ways to add PanDA resource: 
      * from main page click "Define PANDA resource" link
      * from description of PanDA site page click "Add PANDA resource for this PANDA site..." 
      * only five parameters should be defined and only two of them obligatory:
         * PANDA Site (obligatory parameter, start typing and possible variants will be advised)
         * Name of PANDA resource (obligatory parameter)
         * CVMFS 
         * Default analysis 
         * Default production
      * After defining the parameters you should click "Check input data", some additional info will appear.  If the input data are acceptable "Save PANDA resource" button will appear.
         Data will be saved only after clicking of  "Save PANDA resource" button. 

   $ *Update* : To edit PanDA resource description you should click "Update PANDA resource info" link on PanDA resource information page. 
      * Update PanDA resource forms has similar views as the Add one (above), only five parameters should be defined and only two of them obligatory
      * After defining of needful parameters you should click "Check input data", some additional info will appear.  If inputted data acceptable "Save PANDA resource" button will appear.
        Data will be saved only after clicking of  "Save PANDA resource" button. 

   $ *Delete* : click the button "Delete PANDA Resource" in the "update" page (see above)


---++ Add/Clone/Edit/Delete PANDA Queue

Modification of PanDA Queue information is very common and often operation. To add new PanDA Queue you should click "Define PANDA queue" link on AGIS WebUI main page.
After this form of definition of PanDA queue will be open, only couple of parameters in "PanDA queue" definition in AGIS are mandatory, but total number of parameters are huge.  

To provide some advice with filling form, some features was implemented: "example",  "clone". If you already know, that new PanDA queue will be similar with one which already existing you can type name of this queue in 'Specify PANDA Queue' field (autocomplete will give some variants, just start typing). After this, if you will click on "Show as example"  button - description of selected PanDA queue will fill right side of form.  If you will click 'Clone' button - you will be redirected to clone form: left side of form will be filled with values of   selected queue and same values will be shown on right side as example.

There are no additional validation step for creation of PanDA queue, but nothing will be saved if data incomplete or wrong.  After successful saving of PanDA queue description you will be redirected to "PanDA Queue edit" form. On this form you will be able to modify parameters or manage PanDA queue - CE queue relation


   $ *remove a panda queue* : To delete PanDA queue you should:
      * find PanDA queue in PanDA queues list (Section 'PANDA queues' +  filtering)
      * click PanDA queue name (left column)
      * click 'Update PANDAQueue and CE associated info' link
      * click 'Delete PANDA queue' button (bottom right side of page) 
      * confirm deletion (Alert box will appear)
   * removal of a panda queue will be propagated to
      * panda scheadconfig
      * pilot factory config


   $ *Parameters that cannot be changed in AGIS* : <verbatim>
comment, multicloud, nqueue, space, tspace, status, status_control, status_override,
</verbatim>
   * Consult [[SchedconfigParameterDefinitions]] for more details


---++ Rules for adding DDM endpoints to PanDA queues as of January 2018

PanDA server components started using the AGIS "astorages" field end of 2017 to increase the flexibility associating DDM endpoints to PanDA queues. An overview of the possibilities is given in the image below. 
   * You can attach ddm endpoints for input/read ("read_lan") belonging to different sites. The endpoint with "read_lan0" will get the potential input file transfers for the jobs. 
      * The pilot will use one of the attached input endpoints (it will not use other endpoints!) and it is the Rucio list replica function that orders them by closeness.
   * Output endpoints have to belong to the same site and there can only be one endpoint of each spacetoken type (e.g. DATADISK, SCRATCHDISK, etc.). The endpoint with "write_lan0" will be the default endpoint for writing the output.
   * Input and output endpoint do not have to belong to the same site.
 
     <img src="%ATTACHURLPATH%/Screen_Shot_2018-01-29_at_12.48.32.png" alt="Screen_Shot_2018-01-29_at_12.48.32.png" width="718" height="377" /></verbatim>


---++ Associate PandaQueue with CE Queue
Before AGIS the information on which CE (gatekeeper+localqueue) should have been used for a pandaqueue was stored in the "gatekeeper" and in "localqueue" attribute. Those attributes are still in the schedconfig visualization, but they are *not* used anymore.
Wit AGIS now you may associate multiple CEs to the same pandaqueue. We recommend you to make just one pandaqueue for each PandaResource(PandaSiteID), and associate to this pandaqueue many CEs.
To associate PanDA queue and CE queue you should make next steps:
   
   * *Find or define PanDA queue*. To find PanDA queue you should navigate to 'PANDA Queue' section of !WebUI. (e.g. from http://atlas-agis.cern.ch/agis/atlassite/main/Australia-ATLAS/ click on the left menu on Panda (within ATLAS Info)) You will see the list of PanDA queues which can be sorted by different parameters (PanDA queue name, PanDA Site, ATLAS Site etc.) 
   *  *Navigate to PanDA queue description page*: click to PanDA queue name in table (e.g. http://atlas-agis.cern.ch/agis/panda_queue/detail/ANALY_AUSTRALIA/full/ )
   *  *Navigate to PanDA queue editing page*: click "Update PANDAQueue and CE associated info" link
   *  In bottom of page you will find "Associated CE/Queue" section. This section will represent information about already associated CE queues with possibility of manage relations.
   *  To add new association you should click 'Find and associate another CE/Queue' click
   *  Search form will appear. This form needed to restrict list of possible CE queues.  Please, enter name (or part of name) of CE, site or queue  and click 'Search' button 
   *  A list of queues satisfying will appear
   *  To create the link between PanDA queue and CE queue, please check CE queue and click 'Save' 

---++ Add/Edit Computing Element service and CE queue

In most of cases information about services (CE, SE, etc.) collected by AGIS automatically from GOCDB, OIM and other sources, but in exceptional cases, such information can be added using the
WebUI.

To add new CE into AGIS you should:
  
   * Click 'Define CE service' link  on main page of AGIS WebUI
   * Fill form 'Computing Element Creation'
      *  Field 'OIM/GOCDB site' extended with autocomplete - start typing and variants will advised.
      *  'AGIS CE name' - unique name of service in AGIS, usually looks like [Site name]-CE-[endpoint]
      *  'Endpoint' - hostname and port [hostname]:[port]
      *  'Job manger', 'Flavour', 'Version' - parameters, which describe job management software of CE
   *  Click "Check input data" button, in case data acceptable "Save & continue" button appear.
   *  Click "Save & continue" - only after this data will be saved.

After creation of CE you will be redirected to "Computing Element Editing" form, which is similar to creation, but have two extra fields - 'State' and 'State comment'. 

To edit CE service information you should: 
    
   * find service: 'Service' section of WebUI + some filtering will help you
   * Click Service name: 'Computing element description' page will appear
   * Click 'Detailed info' (top right)
   * Click 'Edit CE description':   "Computing Element Editing" form will appear  

To add CE queue to Computing element you should:

   * find service: 'Service' section of WebUI + some filtering will help you
   * Click Service name: 'Computing element description' page will appear
   * Click 'Detailed info' (top right)
   * Click 'Add Queue' link:   "Queue Creation" form will appear

"CE" field of form will be pre filled with correct value. You should fill only 'Queue name' value.  

---++ Add/Edit LFC service (OBSOLETE)

To create new LFC service click "Define LFC service" link on main page of AGIS WebUI.

According to the appeared form, next values should be filled:
   *   *OIM/GOCDB Site*: select from list of sites, known by AGIS
   *   *ATLAS Service Name*: specify some short name of service which will be used to present the service in AGIS WebUI
   *   *Endpoint*: hostname of machine with lfc:// prefix and valid path :/grid/atlas postfix

WebUI will not allow to add inconsistent information to AGIS.

Form for editing information is similar to the form for creation, and have two additional fields: State and State comment.

---++ Add/Edit Storage Element service
To create new Storage Element service click "Define SE service" link on main page of AGIS WebUI.

According to the appeared form, next values should be filled:
   *   *OIM/GOCDB Site*: select from list of sites, known by AGIS
   *   *Endpoint*: hostname of machine
   *   *SE type*: type of storage element should be selected
   *   *Redirector*: if SE type XROOTD or HTTP is selected, it will be possible to define Redirector for the service

WebUI will not allow to add inconsistent information to AGIS. If information is correct, AGIS unique service name will be generated automatically and will appear on the second page.

Form for editing information is similar to the form for creation, and have two additional fields: State and State comment.

---++ Add/Edit Redirector service
To create new Redirector service click "Define Redirector" link on main page of AGIS WebUI.

According to the appeared form, next values should be filled:
   *   *OIM/GOCDB Site*: select from list of sites, known by AGIS
   *   *Endpoint*: hostname of machine with prefix and port
   *   *Parent redirector*: if redirector is being added has parent redirector, it should be selected from the dropdown list, if not, leave it blank
   *   *Protocol*

WebUI will not allow to add inconsistent information to AGIS. If information is correct, AGIS unique service name will be generated automatically and will appear on the second page.

Form for editing information is similar to the form for creation, and have two additional fields: State and State comment.


---++ REST API to (bulk) update PandaQueue object properties

To bulk update the PandaQueue attributes one should POST JSON encoded input (for BULK operation) or apply  GET request (for simple update) to the following URI:
<verbatim>https://atlas-agis-api.cern.ch/request/pandaqueue/update/set_attributes/?json</verbatim>

json_pretty=True flag can be passed to show results in human readable JSON format.

An example of BULK update using curl:

<verbatim>
    curl --capath /etc/grid-security/certificates --cacert $X509_USER_PROXY --cert $X509_USER_PROXY -X POST -d @pq_setattributes.json "https://atlas-agis-api.cern.ch/request/pandaqueue/update/set_attributes/?json&json_pretty=True" > pq_setattributes_out.json
</verbatim>

where input __pq_setattributes.json__ file contains the list of dict entries with defined _pandaqueue object attributes_  to be updated

e.g.
<verbatim>
[
{"name": "AGLT2_MCORE-condor", "capability":"mcore", "corecount":8, "direct_access_wan":true, "description":"new desc value here"},
{"name": "BAD_NAME", "capability":"mcore"}
]
</verbatim>

Server responses with the list of entries like this:
<verbatim>
[ PandaQueue name, return_val, "error message in case of  failure (when return_val is None)" ]
</verbatim>

in case of real DB update the return_val is True, otherwise False means no DB modifications

in out example the response was:
<verbatim>
cat  pq_setattributes_out.json
[
  [
    "AGLT2_MCORE-condor",
    false,
    ""
  ],
  [
    "BAD_NAME",
    null,
    "(agis.utils.exceptions.NonExistsException) Failed to resolve PandaQueueObject by query: {'name': u'BAD_NAME'}, not found!"
  ]
]
</verbatim>

Required input values  to be passed as a dict:
<verbatim>['name']</verbatim> 



*REST API Summary*:

   * API format:
      * update URI: <verbatim>https://atlas-agis-api.cern.ch/request/ddmendpoint/update/set_protocols/?json&</verbatim>
      * Single update:
         * Method: GET
         * Required input to be passed: _name or panda_resource_ plus the list of attributes which need to be updated
      * Bulk update:
         * Method: POST
         * Input: JSON encoded list of dict entries  same as for GET request
         * Server response:
            * JSON encoded list of entries [name, status, “error_message in case of failure”]
            * status=True if case of real db modification
   * Real example:
      <verbatim>https://atlas-agis-api.cern.ch/request/pandaqueue/update/set_attributes/?json&json_pretty=1&name=ANALY_BNL_MCORE-condor&capability=mcore&description=multicore+analysis+queue</verbatim>
   * <verbatim>curl --capath /etc/grid-security/certificates --cacert $X509_USER_PROXY --cert $X509_USER_PROXY -X POST -d @data.json
"https://atlas-agis-api.cern.ch/request/pandaqueue/update/set_attributes/?json" > results.json</verbatim>

In case of wrong attribute passed, full list of accepted parameters will be exposed in error message.

By default, if *_on flag (indicating if property value should be inherited from parent PQ object or not) is not passed to the input, it will be triggered to ON once corresponding PQ.attribute is updated. 

PQ data can be fetched and checked from the PandaQueue schedconfig JSON export: <verbatim>http://atlas-agis-api.cern.ch/request/pandaqueue/query/list/?json&preset=schedconf.all</verbatim>


---++ REST API to (bulk) update DDMEndpoint protocols

To bulk update the protocols data one should POST JSON encoded input to the following URI:
<verbatim>https://atlas-agis-api.cern.ch/request/ddmendpoint/update/set_protocols/?json</verbatim>

an example using curl:

<verbatim>
    curl --capath /etc/grid-security/certificates --cacert $X509_USER_PROXY --cert $X509_USER_PROXY -X POST -d @protocols.json "https://atlas-agis-api.cern.ch/request/ddmendpoint/update/set_protocols/?json" > set_protocols_out.json
</verbatim>

where input __protocols.json__ file contains the list of dict entries with defined _ddmendpoint, se, activity, priority, endpoint, is_default_  keys

e.g.
<verbatim>
[
{"ddmendpoint": "AGLT2_LOCALGROUPDISK", "se":"https://head01.aglt2.org:2880", "activity":"rwd", "priority":1, "endpoint":"/atlaslocalgroupdisk/rucio/"},
{"ddmendpoint": "AGLT2_DATADISK_BAD", "se":"srm://head01.aglt2.org:8443/srm/managerv2?SFN=", "activity":"s", "priority":1, "endpoint":"/test/hhh"}
]</verbatim>

The activity parameter takes either single value ("r") or the list of activities ("rwd")

Server responses with the list of entries like this:
<verbatim>
[ (DDM, SE, activity), return_val, "error message in case of  failure (when return_val is None)" ]
</verbatim>

in case of real DB update the return_val is True, otherwise False means no DB modifications

in out example the response was:
<verbatim>
cat set_protocols_out.json

[
  [
    [
      "AGLT2_LOCALGROUPDISK",
      "https://head01.aglt2.org:2880",
      "r"
    ],
    true,
    ""
  ],
  [
    [
      "AGLT2_LOCALGROUPDISK",
      "https://head01.aglt2.org:2880",
      "w"
    ],
    true,
    ""
  ],
  [
    [
      "AGLT2_LOCALGROUPDISK",
      "https://head01.aglt2.org:2880",
      "d"
    ],
    true,
    ""
  ],
  [
    [
      "AGLT2_LOCALGROUPDISK",
      "https://head01.aglt2.org:2880",
      "d"
    ],
    null,
    "(agis.utils.exceptions.NonExistsException) Failed to resolve DDMEndpoint by name=AGLT2_DATADISK_BAD, not found!"
  ]</verbatim>

the primary key of data entry is {ddm, se, activity}

The deletion of  DDMEndpoint  protocols can be done in the same way
by posting  JSON encoded  data to the URI:
<verbatim>    https://atlas-agis-api.cern.ch/request/ddmendpoint/update/remove_protocols/?json</verbatim>

Required input values  to be passed as a dict:
<verbatim>['ddmendpoint', 'se', 'activity']</verbatim> 



*REST API Summary*:

   * API format:
      * Add URI: <verbatim>https://atlas-agis-api.cern.ch/request/ddmendpoint/update/set_protocols/?json&</verbatim>
      * Remove URI: <verbatim>https://atlas-agis-api.cern.ch/request/ddmendpoint/update/remove_protocols/?json&</verbatim>
      * Single update:
         * Method: GET
         * Required input to be passed: _ddmendpoint, se, activity, priority, endpoint_
      * Bulk update:
         * Method: POST
         * Input: JSON encoded list of dict entries __{'ddmendpoint', 'se', 'activity', 'priority', 'endpoint'}__
         * Server response:
            * JSON encoded list of entries [(ddm,se,activity), status, “error_message in case of failure”]
            * status=True if case of real db modification
   * Real example:
      <verbatim>https://atlas-agis-api.cern.ch/request/ddmendpoint/update/set_protocols/?
json&ddmendpoint=AGLT2_DATADISK&se=srm://head01.aglt2.org:8443/srm/managerv2?
SFN=&activity=rwd&priority=3&endpoint=/pnfs/aglt2.org/atlasdatadisk/rucio/</verbatim>
     * <verbatim>curl --cert $X509_USER_PROXY --key $X509_USER_PROXY -kS -X POST -d @protocols.json
"https://atlas-agis-api.cern.ch/request/ddmendpoint/update/set_protocols/?json" > set_protocols.json</verbatim>


Protocols data can be fetched from DDMEndpoint JSON export: <verbatim>http://atlas-agis-api.cern.ch/request/ddmendpoint/query/list/?json</verbatim>



---++ REST API to (bulk) update PandaQueue status (PanDA blacklisting)

AGIS provides REST API to (bulk) update the blacklisitng information.
it's possible to do it either via sending HTTP POST requests (more convenient for bulk operations) as well as sending HTTP GET request

To bulk update pandaqueue blacklisting statuses one should POST JSON encoded input to the following URI:
<verbatim>https://atlas-agis-api.cern.ch/request/pandaqueuestatus/update/set_probestatus/?json</verbatim>

an example using curl:

<verbatim>
    curl --cert $X509_USER_PROXY --key $X509_USER_PROXY -kS -X POST -d @blacklisting.json "https://atlas-agis-api.cern.ch/request/pandaqueuestatus/update/set_probestatus/?json" > output.json
</verbatim>

where input __blacklisting.json__ file contains the list of dict entries with defined _pandaqueue, probe, value, activity, reason, expiration_  keys

e.g.
<verbatim>
[
{"pandaqueue": "AGLT2_TEST-condor", "activity":"a", "value":"BROKEROFF", "reason":"HC changed status", "probe":"hummercloud", "expiration":"2016-06-07T12:00:00"},
{"pandaqueue": "ANALY_MCGILL",  "value":"TEST", "reason":"HammerCloud PFT test suite", "probe":"hummercloud", "expiration":"2016-06-08T12:00:00"},
]</verbatim>

The activity parameter can be omitted. 

Server responses with the list of entries like this:
<verbatim>
[ (PandaQueue, activity), return_val, "error message in case of  failure (when return_val is None)" ]
</verbatim>

in case of real DB update the return_val is True, otherwise False means no DB modifications

Single update operation can be send using HTTP GET query:

<verbatim>
https://atlas-agis-api.cern.ch/request/pandaqueuestatus/update/set_probestatus/?json&pandaqueue=AGLT2_TEST-condor&value=BROKEROFF&reason=TEST&expiration=2016-06-07T12:00:00&probe=hammercloud&activity=a
</verbatim>


*REST API Summary*:

   * API format:
      * Add URI: <verbatim>https://atlas-agis-api.cern.ch/request/pandaqueuestatus/update/set_probestatus/?json&</verbatim>
      * Single update:
         * Method: GET
         * Required input to be passed: _pandaqueue, value, probe, activity, expiration, reason_
            * accepted values: probe=[`manual`, `hammercloud`, `switcher`], activity=`a`, value = ['ONLINE', 'OFFLINE', 'TEST', 'BROKEROFF, 'AUTO']
            * accepted input: site (name of ATLAS Site, if passed pandaqueue field is ignored) - to blacklist all PandaQueue resource behind given ATLAS Site
      * Bulk update:
         * Method: POST
         * Input: JSON encoded list of dict entries containing required input keys
         * Server response:
            * JSON encoded list of entries [(pandaqueue,activity), status, “error_message in case of failure”]
            * status=True if case of real db modification

   * Real example:
      * Single update
      <verbatim> 
        curl --cert $X509_USER_PROXY --key $X509_USER_PROXY  "https://atlas-agis-api.cern.ch/request/pandaqueuestatus/update/set_probestatus/?json&pandaqueue=AGLT2_TEST-condor&value=BROKEROFF&reason=TEST&expiration=2016-06-07T12:00:00&probe=hammercloud&activity=a"
      </verbatim>
      * Bulk update:
      <verbatim>  
        curl --cert $X509_USER_PROXY --key $X509_USER_PROXY -kS -X POST -d @blacklisting.json "https://atlas-agis-api.cern.ch/request/pandaqueuestatus/update/set_probestatus/?json" > output.json
      </verbatim>


Blacklisting details can be fetched from JSON export: <verbatim>http://atlas-agis-api.cern.ch/request/pandaqueuestatus/query/list/?json</verbatim>
History of Blacklisting details can be fetched from JSON export: <verbatim>http://atlas-agis-api.cern.ch/request/pandaqueuestatus/historty/list/?json</verbatim>

List of accepted keys and possible filters can be resolved from the API doc page: http://atlas-agis-api.cern.ch/request/pandaqueuestatus/query/help/



---+ OLD after the 28th February 2013 -- Temporary instructions during the period in which both AGIS and svn SchedConfig python file are used.
   * [[%ATTACHURL%/20130115_AGIS_PandaSchedConf-Instructions.rtf][20130115_AGIS_PandaSchedConf-Instructions.rtf]]: 20130115_AGIS_PandaSchedConf-Instructions.rtf


<!-- *********************************************************** -->
<!-- Do NOT remove the remaining lines, but add requested info as appropriate -->
<!-- *********************************************************** -->

-----
<!-- For significant updates to the topic, consider adding your 'signature' (beneath this editing box) -->
*Major updates*:%BR%
-- Main.AleDiGGi - 30-Mar-2012
-- Main.DanilaOleynik - 11-Dec-2012
-- Main.DanilaOleynik - 13-Jan-2013
-- Main.DanilaOleynik - 16-Jan-2013

<!-- Person responsible for the page: 
Either leave as is - the creator's name will be inserted; 
Or replace the complete REVINFO tag (including percentages symbols) with a name in the form Main.TwikiUsersName -->
%RESPONSIBLE% %REVINFO{"$wikiusername" rev="1.1"}% %BR%
<!-- Once this page has been reviewed, please add the name and the date e.g. Main.StephenHaywood - 31 Oct 2006 -->
%REVIEW% *Never reviewed*

%STOPINCLUDE%
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
rtf	20130115_AGIS_PandaSchedConf-Instructions.rtf	r1	manage	6.6 K	2013-01-18 - 14:26	AleDiGGi
mov	ATLASSite_create_2.mov	r1	manage	2695.5 K	2012-12-03 - 18:35	DanilaOleynik	ATLAS Site create example.
mov	ATLASSite_edit.mov	r1	manage	3333.0 K	2012-12-03 - 18:40	DanilaOleynik	ATLAS Site edit example.
pptx	Access_protocols_for_DDMEndpoint.pptx	r1	manage	63.8 K	2012-12-10 - 15:38	ArtemPetrosyan
png	Screen_Shot_2018-01-29_at_12.37.46.png	r1	manage	88.9 K	2018-01-29 - 12:38	FernandoHaraldBarreiroMegino
png	Screen_Shot_2018-01-29_at_12.48.32.png	r1	manage	114.5 K	2018-01-29 - 12:49	FernandoHaraldBarreiroMegino
mov	update_ddmendpoint.mov	r1	manage	25872.3 K	2012-09-11 - 19:18	DanilaOleynik	Update DDM Endpoint
Topic revision: r35 - 2019-11-28 - AlexeyAnisyonkov
AtlasComputing
ATLAS Collaboration
Report outdated page
Twiki-Support
Create a LeftBar for this page
- Cern Search
- TWiki Search
- Google Search
Atlas All webs
Copyright &© 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback