FtsTier0ServerInterventionPlanPatch1232 < LCG

LCG Web>FtsWlcg>FtsTier0Deployment>FtsTier0ServerInterventions>FtsTier0ServerInterventionPlanPatch1232 (2007-10-10, SteveTraylen) (raw view)

EditAttachPDF

%INCLUDE{FtsMenu}%


---++ Objective
   * To install [[https://savannah.cern.ch/patch/?1232][Patch #1232]] to the prod and tiertwo service.
   * Reboot for long overdue kernel upgrade.

---++ Status
Completed Thursday August 23rd 2007.

---++ Broadcast
<verbatim>
Date: Thursday August 23rd 2007
Affected Service: CERN FTS prod-fts-ws.cern.ch and tiertwo-fts-ws.cern.ch.
Impact: Service unavailable for atlas, alice, lhcb, cms, dteam an ops VOs.

On Thursday August 23rd from 06:30 UTC (08:30 CEST) the CERN T0 export and tiertwo FTS
service will be unavailable for a software upgrade. The full service is expected to 
be restored by 10:00 UTC (12:00 CEST). No furthur broadcast will be sent assuming
the service is restored by this time.

Questions: fts-support@cern.ch.
</verbatim>

---++ Preparation Steps
   * Check defragmentation of Database - Request made to DBAs.
   * Prepare CDB templates and validate. Same as pilot plus addition of yaim 4.

---++ Intervention
   * Set hosts to SMS status maintenance. *Done*
   * Switch channels to inactive, wait to drain. *Done*

   * Stop scripts running on =fts102= and =lxb2091=
   * Disable history pack on prod and tiertwo.
      * exec fts_stats.stop_hourly_job;
      * exec fts_history.stop_job;
      * exec fts_statecount.stop_job. 
   * Verify no queries are running.
      * select * from user_jobs;
   * Stop the web services.
   * Stop the agent nodes.
   * Ask DBAs to backup tables and check fragmentation.
   * Upgrade the schema.
   * Commit CDB files.
   * Update RPMS.
   * Run yaim.
      * For some unknown reason =/opt/glite/yaim/bin//yaim -i -s /etc/lcg-quattor-site-info.def -m glite-FTA2= works but 
  the more correct. =ncm-ncd --configure yaim= fails. This is not a problem and can be easily worked around.
      * In fact fixed by replacing =tail_pid=$?= rather than =tail_pid=$!=. In fact this yaim is not released and is fixed in what will be release.
   * Start web servers by rebooting.
      * Try a few commands.
   * Start agent nodes by rebooting.
      * Check they are running.
   * Start monitoring jobs =fts102=
   * Enable jobs
      * exec fts_history.submit_job;
      * exec fts_stats.submit_job;
      * exec fts_statecount.submit_job; 
   * Enable production status for nodes in SMS.
   * Have lunch.
     

-- Main.SteveTraylen - 15 Aug 2007

Topic revision: r6 - 2007-10-10 - SteveTraylen

LCG Wikis

LCG Service Coordination

LCG Grid Deployment

LCG Applications Area

Public webs

Welcome Guest

- Cern Search
- TWiki Search
- Google Search
LCG All webs