Technical procedures for incubators during ROC start-up

Links to Documentation

to: the future ROC Manager
Subject: start-up of new ROC *Name*
Body:

I am in charge to follow the start-up of the new ROC NAME on behalf of Incubator NAME. Very soon you will receive an invitation for a kick-off meeting where the initial planning will be discussed. Please let me know if I need to include other recipients in this loop

In order to be prepared at best for the discussion I would ask you to check the following page

https://twiki.cern.ch/twiki/bin/view/EGEE/OperationalDocumentation

The page points to documents you will need to master in order to work as ROC and ROC-on-duty . It would be good if before starting the training-on-the-job the future operators had completed the reading.

In particular I would suggest to follow this order to start

• Operational Procedures
  Operational Procedures for ROD
• Operations Dashboard HOW TO and training
  https://edms.cern.ch/document/1015741
• Security Incident Handling and Response Guide
  https://edms.cern.ch/file/867454/2/EGEE_Incident_Response_Procedure.pdf
• Operations Best Practices
  Operations Best Practices
• Operational Procedures*
  Operational Procedures for ROCs and Sites

And then go on in the order you prefer with the others.

Thanks.

Regards,

your name

Enabling the new ROC over the various Ops tools

There is currently a constraint for this operation represented by the schedule of the GGUS releases. In fact due to many dependencies all the operational tools have to be re-configured in a nearly synchronous way, which can happen only in proximity of a GGUS release, which is rigidly scheduled once per month (normally the last week of the month)

This operation is synchronised using a GGUS ticket as a token. The GGUS ticket should be opened by the incubator conveniently earlier than the GGUS release (e.g. three weeks in advance). The recommendation is to open it as soon as possible asking for the first assignees (the GOCDB) to start at a given date. The template for the GGUS ticket to open follows:

Template of GGUS ticket to request the configuration of the ROC on the Ops tools

Subject: Enabling ROC NAME on Ops tools
Type of problem: Operations
Initial assignment: GOCDB
CC: cic-information@in2p3.fr, project-eu-egee-sa1-cod-pole2-proc@cern.ch
Body:

This ticket is used to track the enabling the new ROC NAME in the EGEE Operational tools.

The request is opened by the ROC incubator (INCUBATOR name) and will be managed 
with this single GGUS ticket moved across the relevant support units impacted by the process.

The required actions are:

A - Creation of a new support unit in GGUS : ROC NAME
B - Creation of a new ROC entry in GOCDB (with no site attached).
     ROC name: NAME (or ROC_NAME if required by new naming conventions)
     ROC management mailing list: xxx@yyy.zzz
     ROC manager: Name, Surname (DN)
C - Configuration of the new entry in CIC dashboard
D - Configuration of the new entry in SAM/nagios
E - Configuration of the new entry in SAMAP
F - Changes in the Operations manual
G - Final confirmation that the new ROC can start the operations
H - transfer of the sites in GOCDB
I - Manual configuration of sites in SAM/Gridview
L - Final checks by INCUBATOR ROC

This ticket should be assigned in the order:

1) To GOCDB to perform B
4) To CIC Portal to perform C
2) To SAM/SFT to perform D
3) To ROC_CE to perform E
5) To GGUS to perform A
6) To C-COD to perform F
7) To ROC INCUBATOR to perform F
8) TO GOCDB to perform H
9) To SAM/SFT to perform I
10) To ROC INCUBATOR to perform final checks (L)

As GGUS is constrained to apply the configuration with the release of the dd-Mon-YY , 
the assignment to GGUS needs to be done at latest on the  (dd-2)-Mon-YY. 
Therefore if that date is approaching the current support unit should re-assign this ticket immediately 
to GGUS (Step 5). Possibly missing steps will be recovered later on.

Thanks,

Regards,

your_namefor the ROC INCUBATOR

Request SA1 Management to insert new ROC in mailing lists

The request is moved to OCC (current contact point for SA1 management)

TO: occ-grid-support@cern.ch
CC: new roc e-mail
Subject : New ROC name to be configured in management e-mail list
Body

Dear SA1 coordinator(s),

The new ROC name is about to start operations:

please add the contact e-mail address 
xxx@eee.sss

to the ROC Managers mailing list 
project-egee-roc-managers@cern.ch

Thanks in advance.

Best regards,

your_namefor the ROC INCUBATOR

Final Checks (after creation, before announcement)

• VOMS groups created
• mailing lists communicated to SA1 management
• GOCDB entry completely defined
• test of the operational tools
• End-to-end tests of GGUS and ROD workflows

Announcing the new ROC in operations

Broadcast Tool in the CIC Portal

Template for announcing the new ROC in operations (using the CIC Portal broadcast tool)

TO: Publish in the CIC Portal news, CIC-on-Duty, OSG, ROC managers, VO managers, support-eis@cern.ch

SUBJECT:

 ROC name is operational

MESSAGE:

Dear members of the EGEE Grid Production Service,

This is to announce to all the concerned parties that the new ROC  name is now operational and fully integrated with the production infrastructure.

With the previous agreement of the site Managers the responsibility for the following sites has been moved 
at all practical effects from ROC incubator to the new ROC.
 - SITE_NAME (Real Name)
 - SITE_NAME (Real Name)
 - SITE_NAME (Real Name)
 - SITE_NAME (Real Name)
 - SITE_NAME (Real Name) - Uncertified
 - SITE_NAME (Real Name) - Uncertified 

The new operational structure is already reflected in all the operational tools.

Best wishes to our colleagues of ROC Canada for the continuation of their work in EGEE Operations.

Best regards,
   Your Name on behalf of  the ROC  name