CRAB Server on Docker + Kubernetes

Complete: 5 Go to SWGuideCrab

how to build a CRABSERVER container for CMSWEB K8s cluster

If you are not belforte just replace your username, favorite directories and docker hub username as appropriate

1. Build RPMs (as usual)

quick recap from instructions in https://twiki.cern.ch/twiki/bin/view/CMSPublic/NotesAboutReleaseManagement#Releasing_a_CRAB_TaskWorker_rele

  • go to the build node
    ssh vocms055
    cd /buid/belforte/cmsdist
    

  • git fetch/merge/update as needed

  • now change crabserver(cache/taskworker) and wcore version as needed/usual
    • vim crabserver.spec

  • build and upload (before uploading RPMs, make sure that you are subscribed to cms-comp-devs e group which will give you required permissions to upload it)
    cd /build/belforte
    pkgtools/cmsBuild -c cmsdist --repository comp -a slc7_amd64_gcc630 --builders 8 -j 5 --work-dir w build comp
    pkgtools/cmsBuild -c cmsdist --repository comp -a slc7_amd64_gcc630 --builders 8 -j 5 --work-dir w --upload-user=$USER upload comp
    

2. Build container (replaces the installation on VM)

  • go to the build node (ask Shahzad for access)
    ssh cmsdocker01
    

  • git fetch/merge/update as needed
  • now change VER and REPO e.g .to VER=HG1911a-comp and REPO=comp.belforte
    vim crabserver/install.sh
    

There are two ways to build docker image:

1. You can manually run docker commands

  • build image
    docker build -t sbelforte/crabserver:v3.210301 crabserver
    

  • push to docker
    docker push sbelforte/crabserver:v3.210301
    

  • cleanup(IMAGE ID is from docker images )
    docker rmi <IMAGE ID>
    

2. Use build.sh script that will build, push, delete image

  • run script
    CMSK8STAG=v3.210301 ./build.sh "crabserver"
    
However, be aware that if you use this script, image will be build and uploaded to cmssw DH account. If you want to upload to your private repo, you have to do it manually.

  • check with e.g.
    docker images
    docker inspect sbelforte/crabserver:v3.210301
    docker inspect <IMAGE ID> # where <IMAGE ID> is from docker images output
    docker run --rm -h `hostname -f` -v /tmp:/tmp -i -t sbelforte/crabserver:v3.210301 /bin/bash
    
    • the latter command logs you in the container and where can examine the /data/srv tree

3. Use K8S to run this container

we have access to deploy CRAB services to 3 environments:

  • dev cluster
  • testbed cluster
  • production cluster

Depending on where you need to deploy your service, you need to use proper KUBECONFIG configuration. Below example how to deploy to test cluster is provided. Details how to access testbed can be found here: https://cms-http-group.docs.cern.ch/k8s_cluster/cmsweb_testbed_cluster_doc/ and production cluster here: https://cms-http-group.docs.cern.ch/k8s_cluster/cmsweb_production_cluster_doc/. Service deployment details are provided here: https://cms-http-group.docs.cern.ch/k8s_cluster/deploy-srv/

Development Kubernetes clusters

In total 5 clusters are available for developers to test their services which are listed below:
  • cmsweb-test1.cern.ch
  • cmsweb-test2.cern.ch
  • cmsweb-test3.cern.ch
  • cmsweb-test4.cern.ch
  • cmsweb-test5.cern.ch

Each of these clusters can be accessed by setting

  • export KUBECONFIG=/afs/cern.ch/user/m/mimran/public/cmsweb-k8s/config.cmsweb-test1
  • export KUBECONFIG=/afs/cern.ch/user/m/mimran/public/cmsweb-k8s/config.cmsweb-test2
  • export KUBECONFIG=/afs/cern.ch/user/m/mimran/public/cmsweb-k8s/config.cmsweb-test3
  • export KUBECONFIG=/afs/cern.ch/user/m/mimran/public/cmsweb-k8s/config.cmsweb-test4
  • export KUBECONFIG=/afs/cern.ch/user/m/mimran/public/cmsweb-k8s/config.cmsweb-test5
  • export KUBECONFIG=/afs/cern.ch/user/m/mimran/public/cmsweb-k8s/config.cmsweb-test6

Cluster number 2 is dedicated for and should be used by CRAB team. Depending on which cluster you want to access, you will use different KUBECONFIG value.

Access to the cluster

  • kubernetes tools are deployed at CERN on CC8 machines
    ssh lxplus8
    

  • setup OpenStack Project, OS_TOKEN and cluster's OpensStack confing as per Valentin's instructions. I.e. via
    export OS_PROJECT_NAME="CMS Webtools Mig"
    export KUBECONFIG=/afs/cern.ch/user/m/mimran/public/cmsweb-k8s/config.cmsweb-test2
    

  • get pods and verify that your setup is correct, you point to right cluster etc.
    kubectl -n crab get pod
    

Deploy new crabserver version

  • go to the directory where you cloned CMSKubernetes
    cd ~/WORK/GIT/CMSKubernetes/kubernetes/cmsweb
    

  • git fetch/merge/update as needed
  • use deploy-srv.sh script to deploy service
    ./scripts/deploy-srv.sh crabserver v3.210301 test
    

  • check
    kubectl -n crab exec -ti {POD_ID} bash
    

  • check that /data/srv/current points to correct version
    ls -l /data/srv/current
    ls  /data/srv/current/sw/slc7_amd64_gcc630/cms/crabserver/
    

Remove running crabserver service

kubectl -n crab delete deployment crabserver

Create a new crabserver pod

  • Clone repo
    git  clone https://github.com/dmwm/CMSKubernetes.git
    

  • Goto the services directory, which contains all .yaml files
    cd CMSKubernetes/kubernetes/cmsweb/services
    

  • Create/apply crabserver pod with the help of corresponding .yaml file
    kubectl -n crab apply -f crabserver.yaml  --validate=false
    
    service/crabserver created
    configmap/crabserver created
    deployment.apps/crabserver created
    

Restart crabserver pod

Or, better stated, how to restart all pods which run a given service, i.e. a Deployment (see yaml files) which may indicate several pods running the same service for load share/balance If no change to yaml file is needed, use this trick from https://medium.com/faun/how-to-restart-kubernetes-pod-7c702ca984c1 :

kubectl -n crab scale deployment crabserver --replicas=0
# followed later by
kubectl -n crab scale deployment crabserver --replicas=1 # or whatever
e.g.
belforte@lxplus735/services> kubectl -n crab get pod
NAME                          READY   STATUS      RESTARTS   AGE
crabcache-686d5b9ffc-q47lq    1/1     Running     0          2d10h
crabserver-85688cfbdb-pd4kl   1/1     Running     0          2d10h
cron-proxy-1587773100-gmcpc   0/1     Completed   0          2d21h
cron-proxy-1587859500-h96m7   0/1     Completed   0          45h
cron-proxy-1587945900-drmg5   0/1     Completed   0          21h
belforte@lxplus735/services>
belforte@lxplus735/services> kubectl -n crab scale deployment crabserver --replicas=0
deployment.apps/crabserver scaled
belforte@lxplus735/services> kubectl -n crab get pod
NAME                          READY   STATUS        RESTARTS   AGE
crabcache-686d5b9ffc-q47lq    1/1     Running       0          2d10h
crabserver-85688cfbdb-pd4kl   1/1     Terminating   0          2d10h
cron-proxy-1587773100-gmcpc   0/1     Completed     0          2d21h
cron-proxy-1587859500-h96m7   0/1     Completed     0          45h
cron-proxy-1587945900-drmg5   0/1     Completed     0          21h
belforte@lxplus735/services> 
# after a minute or so
belforte@lxplus735/services> 
belforte@lxplus735/services> kubectl -n crab get pod
NAME                          READY   STATUS      RESTARTS   AGE
crabcache-686d5b9ffc-q47lq    1/1     Running     0          2d10h
cron-proxy-1587773100-gmcpc   0/1     Completed   0          2d21h
cron-proxy-1587859500-h96m7   0/1     Completed   0          45h
cron-proxy-1587945900-drmg5   0/1     Completed   0          21h
belforte@lxplus735/services> 
belforte@lxplus735/services> kubectl -n crab scale deployment crabserver --replicas=1
deployment.apps/crabserver scaled
belforte@lxplus735/services> kubectl -n crab get pod
NAME                          READY   STATUS      RESTARTS   AGE
crabcache-686d5b9ffc-q47lq    1/1     Running     0          2d10h
crabserver-85688cfbdb-hmjjm   1/1     Running     0          6s
cron-proxy-1587773100-gmcpc   0/1     Completed   0          2d21h
cron-proxy-1587859500-h96m7   0/1     Completed   0          45h
cron-proxy-1587945900-drmg5   0/1     Completed   0          21h
belforte@lxplus735/services> 

Delete crabserver pod

  • Clone repo
    git  clone https://github.com/dmwm/CMSKubernetes.git
    

  • Goto the services directory, which contains all .yaml files
    cd CMSKubernetes/kubernetes/cmsweb/services
    

  • Delete crabserver pod with the help of corresponding .yaml file
    kubectl delete -f crabserver.yaml
    
    service "crabserver" deleted
    configmap "crabserver" deleted
    deployment.apps "crabserver" deleted
    

Operations within crabserver pod

Start crabserver service within pod
sudo -H -u _crabserver bashs -l -c "/data/srv/current/config/crabserver/manage start 'I did read documentation'"
I did read documentation
stopping crabserver
crabserver not running, not killing
starting crabserver
crabserver not running, not killing

Check crabserver status within pod
sudo -H -u _crabserver bashs -l -c '/data/srv/current/config/crabserver/manage status'
crabserver is RUNNING, PID 132

Restart crabserver service within pod
sudo -H -u _crabserver bashs -l -c "/data/srv/current/config/crabserver/manage start 'I did read documentation'"
I did read documentation
stopping crabserver
crabserver not running, not killing
starting crabserver
crabserver not running, not killing

Stop crabserver service within pod
sudo -H -u _crabserver bashs -l -c "/data/srv/current/config/crabserver/manage stop 'I did read documentation'"
stopping crabserver
Killing crabserver pgid 132 .

Edit service deployment

If for some reason you want to edit given service deployment, you can do that by using command:
kubectl edit deployment crabserver -n crab
deployment.apps/crabserver edited

4. Working with secrets

Show all secrets names with basic info within CRAB namespace

kubectl -n crab get secrets
NAME                              TYPE                                      DATA   AGE
crabcache-secrets                   x                                        4      23h
crabserver-secrets                  x                                        6      23h

Print secret's content

kubectl -n crab  get secrets crabserver-secrets -o yaml |grep CRABServerAuth.py:|cut -d ' ' -f4|base64 --decode 

Delete secrets

kubectl -n crab delete secrets crabserver-secrets
secret "crabserver-secrets" deleted

Deploy/create new secrets

  • Clone CMSKubernetes repo and checkout to the proper branch
    git clone https://:@gitlab.cern.ch:8443/cmsweb-k8s/services_config.git
    git checkout preprod
    
  • Clone cmsweb-k8s repo
    git  clone https://github.com/dmwm/CMSKubernetes.git
    
  • If you want to change password and or database instance then create CRABServerAuth.py file with new DB configuration
    cat ~/git_repos/services_config/crabserver/CRABServerAuth.py
    
  • Need the following files, I got these from Imran and put into ~/git_repos/imranFiles
    ll /afs/cern.ch/user/p/prsharma/git_repos/imranFiles
    
    -r--------. 1 prsharma zh 3044 Apr  9 13:41 cmsweb-hostcert.pem
    -r--------. 1 prsharma zh 1828 Apr  9 13:41 cmsweb-hostkey.pem
    -rw-r--r--. 1 prsharma zh   20 Apr  9 13:32 hmac
    -r--------. 1 prsharma zh 3387 Apr  9 13:41 robotcert.pem
    -r--------. 1 prsharma zh 1828 Apr  9 13:31 robotkey.pem
    
  • Go to the cmswb directory and execute the following command to create/deploy new secrets
    ./scripts/deploy.sh create secrets ~/git_repos/services_config/ ~/git_repos/imranFiles ~/git_repos/imranFiles/hmac
    
    ...
    ---
    Create secrets in namespace: crab
    secret/robot-secrets configured
    secret/proxy-secrets configured
    +++ generate cmsweb service secrets
    secret/crabcache-secrets created
    secret/crabserver-secrets created
    ---
    ...
    crab              crabcache-secrets                       x                                4      55s
    ...
    

Note1: If you want to change password and database instance for crabserver, then follow the sequence of the following steps:

  1. Delete secrets;
  2. Deploy/create new secretswith appropriate changes in CRABServerAuth.py file;
  3. Delete crabserver pod;
  4. Create new crabserver pod.

5. Bookkeeping

remember to

-- StefanoBelforte - 2019-10-23

Edit | Attach | Watch | Print version | History: r18 < r17 < r16 < r15 < r14 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r18 - 2021-04-01 - StefanoBelforte
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback