Amazon EC2 Tutorial

First Meeting's Notes

  1. We get a machine at Brown for WMA deployment (condor will be running there)
  2. This machine comes with a service certificate, Sanjay will provide it and then we register it in SiteDB
  3. In principle we do not need root access to deploy WMA there.
  4. stage out to T2_CH_CERN (EOS) and there will be no stage in.
  5. Remove the requirements line from the job jdl
  6. We will initially have access to 2 or 3 nodes via ssh to debug jobs running/not running
  7. In principle users won't need to create requests (nor jobs)
  8. Then we'll do some slides explaining:
    1. Types of workflows
    2. Generalities about WMAgent
    3. Show a workflow running in Amazon EC2

Machine for WMAgent

Located at Brown
url
gls.hep.brown.edu
userId
awsec2
passwd
the same Certificates are located at /etc/grid-security/

  • The machines needed a service certificate in order to connect to cmsweb to pull all the information for deploying.
  • created the standard set of directories:
/data/srv
/data/srv/wmagent
/data/admin
/data/admin/wmagent

  • created the secrets file:
MYSQL_USER=awsec2
MYSQL_PASS=MySqlDeffaultPassword2
COUCH_USER=awsec2
COUCH_PASS=CouchDbDeffaultPassword3
COUCH_PORT=5984
COUCH_HOST=127.0.0.1
COUCH_CERT_FILE=/data/certs/servicecert.pem
COUCH_KEY_FILE=/data/certs/servicekey.pem
#GLOBAL_WORKQUEUE_URL=https://cmsweb-testbed.cern.ch/couchdb/workqueue
#WMSTATS_URL=https://cmsweb-testbed.cern.ch/couchdb/wmstats
#REQMGR_URL=https://cmsweb-testbed.cern.ch/reqmgr/rest
#ACDC_URL=https://cmsweb-testbed.cern.ch/couchdb/acdcserver
#WORKLOAD_SUMMARY_URL=https://cmsweb-testbed.cern.ch/couchdb/workloadsummary
#DBS3_URL=https://cmsweb-testbed.cern.ch/dbs/int/global/DBSWriter
#PHEDEX_URL=https://cmsweb-testbed.cern.ch/phedex/datasvc/json/prod/
#DQM_URL=https://cmsweb-testbed.cern.ch/dqm/dev/
#DASHBOARD_URL=http://dashb-ssb.cern.ch/dashboard
  • Created the env file:
# getting the condor commands
# getting the condor commands
cd /data/srv/wmagent/current

# exporting the variables to the enviroment
export WMAGENT_SECRETS_LOCATION=/data/admin/wmagent/WMAgent.secrets
export X509_HOST_CERT=/etc/grid-security/hostcert.pem
export X509_HOST_KEY=/etc/grid-security/hostkey.pem
export X509_USER_CERT=/data/certs/servicecert.pem
export X509_USER_KEY=/data/certs/servicekey.pem
export X509_USER_PROXY=/data/certs/myproxy.pem
export install=/data/srv/wmagent/current/install/wmagent
export config=/data/srv/wmagent/current/config/wmagent
export manage=$config/manage

alias condorq='condor_q -format "%i." ClusterID -format "%s " ProcId -format " %i " JobStatus  -format " %d " ServerTime-EnteredCurrentStatus -format "%s" UserLog -format " %s\n" DESIRED_Sites'
alias condor_overview='python /data/srv/wmagent/condor_overview.py

  • However without a service certificate we cannot connect to any cmsweb service.
[awsec2@gls srv]$ sh deployProd.sh -w 1.0.0.patch4 -c HG1410d -t ec2 -s slc6_amd64_gcc481 -r comp=comp.pre
/data/admin/wmagent/env.sh: line 2: cd: /data/srv/wmagent/current: No such file or directory

Starting new agent deployment with the following data:
 - WMAgent version: 1.0.0.patch4
 - CMSWEB tag     : HG1410d
 - Team name      : ec2
 - WMAgent Arch   : slc6_amd64_gcc481
 - Repository     : comp=comp.pre
 - Agent number   : 0
 - DB Flavor      : mysql
 - Use /data1     : false

2015-01-28 05:51:25 URL:https://codeload.github.com/dmwm/deployment/zip/HG1410d [785819] -> "deployment.zip" [1]
2015-01-28 05:51:26 URL:https://github.com/amaltaro/scripts/commit/04593133d62e4f8f9c724b9fe43ebe3c8dfe34cd.patch [848] -> "-" [1]
patching file my.cnf
/data/srv/deployment-HG1410d

*** Removing the current crontab ***
no crontab for awsec2
Done!

*** Bootstrapping WMAgent: prep ***
INFO: 20150128055126: starting deployment of: wmagent
INFO: deploying wmagent - variant: default, version: default
INFO: installation log can be found in /data/srv/wmagent/.deploy/20150128-055126-15492-prep.log
INFO: installation completed sucessfully

*** Deploying WMAgent: sw ***
INFO: 20150128055126: starting deployment of: wmagent
INFO: deploying wmagent - variant: default, version: default
INFO: bootstrapping comp.pre software area in /data/srv/wmagent/v1.0.0.patch4/sw.pre

Worker Nodes

EC2 Cluster nodes: ec2-54-68-24-207.us-west-2.compute.amazonaws.com ec2-54-69-22-181.us-west-2.compute.amazonaws.com UserId = ec2-user

-- JulianBadillo - 2015-01-27

Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r3 - 2015-01-28 - JulianBadillo
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback