CRAB Server on Docker + Kubernetes

Complete:

Go to SWGuideCrab

how to build a CRABSERVER container for CMSWEB K8s cluster

If you are not belforte just replace your username, favorite directories and docker hub username as appropriate

1. Build RPMs (as usual)

quick recap from instructions in https://twiki.cern.ch/twiki/bin/view/CMSPublic/NotesAboutReleaseManagement#Releasing_a_CRAB_TaskWorker_rele

• go to the build node

  ssh vocms055
  cd /buid/belforte/cmsdist
  

• git fetch/merge/update as needed

• now change crabserver(cache/taskworker) and wcore version as needed/usual
  • vim crabserver.spec

• build and upload (before uploading RPMs, make sure that you are subscribed to cms-comp-devs e group which will give you required permissions to upload it)

  cd /build/belforte
  pkgtools/cmsBuild -c cmsdist --repository comp -a slc7_amd64_gcc630 --builders 8 -j 5 --work-dir w build comp
  pkgtools/cmsBuild -c cmsdist --repository comp -a slc7_amd64_gcc630 --builders 8 -j 5 --work-dir w --upload-user=$USER upload comp
  

• note the HG version and crabserver etc. version in the rpms
  • e.g. at http://cmsrep.cern.ch/cmssw/repos/comp.belforte/slc7_amd64_gcc630/latest/RPMS.json
  • or with

    ls w/BUILD/slc7_amd64_gcc630/cms/comp/|tail -1
    

    • etc. replacing comp with crabserver, crabcache etc.
  • or all in one shot

    for pkg in `ls w/BUILD/slc7_amd64_gcc630/cms/`; do echo $pkg; ls -lgort w/BUILD/slc7_amd64_gcc630/cms/$pkg/|tail -1; done
    

2. Build container (replaces the installation on VM)

• go to the build node (ask Shahzad for access)

  ssh cmsdocker01
  

• go to the director where you have cloned https://github.com/dmwm/CMSKubernetes (or your fork of it)

  cd ~/WORK/GIT/CMSKubernetes/docker/
  

• git fetch/merge/update as needed
• now change VER and REPO e.g .to VER=HG1911a-comp and REPO=comp.belforte

  vim crabserver/install.sh
  

There are two ways to build docker image:

1. You can manually run docker commands

• build image

  docker build -t sbelforte/crabserver:v3.210301 crabserver
  

• push to docker

  docker push sbelforte/crabserver:v3.210301
  

• cleanup(IMAGE ID is from docker images )

  docker rmi <IMAGE ID>
  

2. Use build.sh script that will build, push, delete image

• run script

  CMSK8STAG=v3.210301 ./build.sh "crabserver"
  

However, be aware that if you use this script, image will be build and uploaded to cmssw DH account. If you want to upload to your private repo, you have to do it manually.

• check with e.g.

  docker images
  docker inspect sbelforte/crabserver:v3.210301
  docker inspect <IMAGE ID> # where <IMAGE ID> is from docker images output
  docker run --rm -h `hostname -f` -v /tmp:/tmp -i -t sbelforte/crabserver:v3.210301 /bin/bash
  

  • the latter command logs you in the container and where can examine the /data/srv tree

3. Use K8S to run this container

we have access to deploy CRAB services to 3 environments:

• dev cluster
• testbed cluster
• production cluster

Depending on where you need to deploy your service, you need to use proper KUBECONFIG configuration. Below example how to deploy to test cluster is provided. Details how to access testbed can be found here: https://cms-http-group.docs.cern.ch/k8s_cluster/cmsweb_testbed_cluster_doc/ and production cluster here: https://cms-http-group.docs.cern.ch/k8s_cluster/cmsweb_production_cluster_doc/. Service deployment details are provided here: https://cms-http-group.docs.cern.ch/k8s_cluster/deploy-srv/

Development Kubernetes clusters

In total 5 clusters are available for developers to test their services which are listed below:

• cmsweb-test1.cern.ch
• cmsweb-test2.cern.ch
• cmsweb-test3.cern.ch
• cmsweb-test4.cern.ch
• cmsweb-test5.cern.ch

Each of these clusters can be accessed by setting

• export KUBECONFIG=/afs/cern.ch/user/m/mimran/public/cmsweb-k8s/config.cmsweb-test1
• export KUBECONFIG=/afs/cern.ch/user/m/mimran/public/cmsweb-k8s/config.cmsweb-test2
• export KUBECONFIG=/afs/cern.ch/user/m/mimran/public/cmsweb-k8s/config.cmsweb-test3
• export KUBECONFIG=/afs/cern.ch/user/m/mimran/public/cmsweb-k8s/config.cmsweb-test4
• export KUBECONFIG=/afs/cern.ch/user/m/mimran/public/cmsweb-k8s/config.cmsweb-test5
• export KUBECONFIG=/afs/cern.ch/user/m/mimran/public/cmsweb-k8s/config.cmsweb-test6

Cluster number 2 is dedicated for and should be used by CRAB team. Depending on which cluster you want to access, you will use different KUBECONFIG value.

Access to the cluster

• kubernetes tools are deployed at CERN on CC8 machines

  ssh lxplus8
  

• setup OpenStack Project, OS_TOKEN and cluster's OpensStack confing as per Valentin's instructions. I.e. via

  export OS_PROJECT_NAME="CMS Webtools Mig"
  export KUBECONFIG=/afs/cern.ch/user/m/mimran/public/cmsweb-k8s/config.cmsweb-test2
  

• get pods and verify that your setup is correct, you point to right cluster etc.

  kubectl -n crab get pod
  

Deploy new crabserver version

• go to the directory where you cloned CMSKubernetes

  cd ~/WORK/GIT/CMSKubernetes/kubernetes/cmsweb
  

• git fetch/merge/update as needed
• use deploy-srv.sh script to deploy service

  ./scripts/deploy-srv.sh crabserver v3.210301 test
  

• check

  kubectl -n crab exec -ti {POD_ID} bash
  

• check that /data/srv/current points to correct version

  ls -l /data/srv/current
  ls  /data/srv/current/sw/slc7_amd64_gcc630/cms/crabserver/
  

Remove running crabserver service

kubectl -n crab delete deployment crabserver

Create a new crabserver pod

• Clone repo

  git  clone https://github.com/dmwm/CMSKubernetes.git
  

• Goto the services directory, which contains all .yaml files

  cd CMSKubernetes/kubernetes/cmsweb/services
  

• Create/apply crabserver pod with the help of corresponding .yaml file

  kubectl -n crab apply -f crabserver.yaml  --validate=false
  

  service/crabserver created
  configmap/crabserver created
  deployment.apps/crabserver created
  

Restart crabserver pod

Or, better stated, how to restart all pods which run a given service, i.e. a Deployment (see yaml files) which may indicate several pods running the same service for load share/balance If no change to yaml file is needed, use this trick from https://medium.com/faun/how-to-restart-kubernetes-pod-7c702ca984c1 :

kubectl -n crab scale deployment crabserver --replicas=0
# followed later by
kubectl -n crab scale deployment crabserver --replicas=1 # or whatever

e.g.

belforte@lxplus735/services> kubectl -n crab get pod
NAME                          READY   STATUS      RESTARTS   AGE
crabcache-686d5b9ffc-q47lq    1/1     Running     0          2d10h
crabserver-85688cfbdb-pd4kl   1/1     Running     0          2d10h
cron-proxy-1587773100-gmcpc   0/1     Completed   0          2d21h
cron-proxy-1587859500-h96m7   0/1     Completed   0          45h
cron-proxy-1587945900-drmg5   0/1     Completed   0          21h
belforte@lxplus735/services>
belforte@lxplus735/services> kubectl -n crab scale deployment crabserver --replicas=0
deployment.apps/crabserver scaled
belforte@lxplus735/services> kubectl -n crab get pod
NAME                          READY   STATUS        RESTARTS   AGE
crabcache-686d5b9ffc-q47lq    1/1     Running       0          2d10h
crabserver-85688cfbdb-pd4kl   1/1     Terminating   0          2d10h
cron-proxy-1587773100-gmcpc   0/1     Completed     0          2d21h
cron-proxy-1587859500-h96m7   0/1     Completed     0          45h
cron-proxy-1587945900-drmg5   0/1     Completed     0          21h
belforte@lxplus735/services> 
# after a minute or so
belforte@lxplus735/services> 
belforte@lxplus735/services> kubectl -n crab get pod
NAME                          READY   STATUS      RESTARTS   AGE
crabcache-686d5b9ffc-q47lq    1/1     Running     0          2d10h
cron-proxy-1587773100-gmcpc   0/1     Completed   0          2d21h
cron-proxy-1587859500-h96m7   0/1     Completed   0          45h
cron-proxy-1587945900-drmg5   0/1     Completed   0          21h
belforte@lxplus735/services> 
belforte@lxplus735/services> kubectl -n crab scale deployment crabserver --replicas=1
deployment.apps/crabserver scaled
belforte@lxplus735/services> kubectl -n crab get pod
NAME                          READY   STATUS      RESTARTS   AGE
crabcache-686d5b9ffc-q47lq    1/1     Running     0          2d10h
crabserver-85688cfbdb-hmjjm   1/1     Running     0          6s
cron-proxy-1587773100-gmcpc   0/1     Completed   0          2d21h
cron-proxy-1587859500-h96m7   0/1     Completed   0          45h
cron-proxy-1587945900-drmg5   0/1     Completed   0          21h
belforte@lxplus735/services> 

Delete crabserver pod

• Clone repo

  git  clone https://github.com/dmwm/CMSKubernetes.git
  

• Goto the services directory, which contains all .yaml files

  cd CMSKubernetes/kubernetes/cmsweb/services
  

• Delete crabserver pod with the help of corresponding .yaml file

  kubectl delete -f crabserver.yaml
  

  service "crabserver" deleted
  configmap "crabserver" deleted
  deployment.apps "crabserver" deleted
  

Operations within crabserver pod

Start crabserver service within pod

sudo -H -u _crabserver bashs -l -c "/data/srv/current/config/crabserver/manage start 'I did read documentation'"

I did read documentation
stopping crabserver
crabserver not running, not killing
starting crabserver
crabserver not running, not killing

Check crabserver status within pod

sudo -H -u _crabserver bashs -l -c '/data/srv/current/config/crabserver/manage status'

crabserver is RUNNING, PID 132

Restart crabserver service within pod

sudo -H -u _crabserver bashs -l -c "/data/srv/current/config/crabserver/manage start 'I did read documentation'"

I did read documentation
stopping crabserver
crabserver not running, not killing
starting crabserver
crabserver not running, not killing

Stop crabserver service within pod

sudo -H -u _crabserver bashs -l -c "/data/srv/current/config/crabserver/manage stop 'I did read documentation'"

stopping crabserver
Killing crabserver pgid 132 .

Edit service deployment

If for some reason you want to edit given service deployment, you can do that by using command:

kubectl edit deployment crabserver -n crab

deployment.apps/crabserver edited

4. Working with secrets

Show all secrets names with basic info within CRAB namespace

kubectl -n crab get secrets

NAME                              TYPE                                      DATA   AGE
crabcache-secrets                   x                                        4      23h
crabserver-secrets                  x                                        6      23h

Print secret's content

kubectl -n crab  get secrets crabserver-secrets -o yaml |grep CRABServerAuth.py:|cut -d ' ' -f4|base64 --decode 

Delete secrets

kubectl -n crab delete secrets crabserver-secrets

secret "crabserver-secrets" deleted

Deploy/create new secrets

• Clone CMSKubernetes repo and checkout to the proper branch

  git clone https://:@gitlab.cern.ch:8443/cmsweb-k8s/services_config.git
  git checkout preprod
  

• Clone cmsweb-k8s repo

  git  clone https://github.com/dmwm/CMSKubernetes.git
  

• If you want to change password and or database instance then create CRABServerAuth.py file with new DB configuration

  cat ~/git_repos/services_config/crabserver/CRABServerAuth.py
  

• Need the following files, I got these from Imran and put into ~/git_repos/imranFiles

  ll /afs/cern.ch/user/p/prsharma/git_repos/imranFiles
  

  -r--------. 1 prsharma zh 3044 Apr  9 13:41 cmsweb-hostcert.pem
  -r--------. 1 prsharma zh 1828 Apr  9 13:41 cmsweb-hostkey.pem
  -rw-r--r--. 1 prsharma zh   20 Apr  9 13:32 hmac
  -r--------. 1 prsharma zh 3387 Apr  9 13:41 robotcert.pem
  -r--------. 1 prsharma zh 1828 Apr  9 13:31 robotkey.pem
  

• Go to the cmswb directory and execute the following command to create/deploy new secrets

  ./scripts/deploy.sh create secrets ~/git_repos/services_config/ ~/git_repos/imranFiles ~/git_repos/imranFiles/hmac
  

  ...
  ---
  Create secrets in namespace: crab
  secret/robot-secrets configured
  secret/proxy-secrets configured
  +++ generate cmsweb service secrets
  secret/crabcache-secrets created
  secret/crabserver-secrets created
  ---
  ...
  crab              crabcache-secrets                       x                                4      55s
  ...
  

Note1: If you want to change password and database instance for crabserver, then follow the sequence of the following steps:

1. Delete secrets;
2. Deploy/create new secretswith appropriate changes in CRABServerAuth.py file;
3. Delete crabserver pod;
4. Create new crabserver pod.

5. Bookkeeping

remember to

• write e-log entry
  • https://cms-logbook.cern.ch/elog/Analysis+Operations/?cmd=New
• make a PR for CMSKubernetes GIT so that it stays up to date on which image should be deployed
  • example: https://github.com/dmwm/CMSKubernetes/pull/80

-- StefanoBelforte - 2019-10-23