Kubernetes Docker for CRAB


  • create cluster in OS web ui with tempate cmsweb-stable, one master (small), one node (default) and call it call it crabdev-frontend
  • ssh lxplus-cloud
  • define some useful vars. first two are used by Valentin's scripts. Others are for my convenience
    export CMSWEB_CLUSTER=crabdev-frontend
    export CMSWEB_KEY=/afs/cern.ch/user/b/belforte/private/belforte-Cloud.pem
    export CRAB_FE_DIR=/afs/cern.ch/user/b/belforte/WORK/K8s/FE
    export CMSWEB_K8s=/afs/cern.ch/user/b/belforte/WORK/GIT/CMSKubernetes/ # this is where I cloned git@github.com:belforte/CMSKubernetes.git
    # the above is a fork of https://github.com/dmwm/CMSKubernetes.git 
    export CMSWEB_CONFIG=/afs/cern.ch/user/b/belforte/WORK/K8s/preprod/ # this is where I cloned https://gitlab.cern.ch/cmsweb-k8s/preprod.git
  • define the OpenStack project export OS_PROJECT_NAME="CMS Webtools Mig"
  • use it to prepare configuration for Kubernetes
    openstack coe cluster list
    mkdir -p ${CRAB_FE_DIR}
    cd ${CRAB_FE_DIR}
    openstack coe cluster config crabdev-frontend
    export KUBECONFIG=${CRAB_FE_DIR}/config
  • check
    kubectl get node
    CRAB_FE_MASTER=`kubectl get node|grep master|awk '{print $1}'`
    CRAB_FE_MINION=`kubectl get node|grep minion|awk '{print $1}'`   # my cluster only has one minion
    ssh -i $CMSWEB_KEY -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no fedora@$CRAB_FE_MINION
  • prepare area for certs
    mkdir -p $CRAB_FE_CERTS
    cp ~/.globus/usercert.pem  $CRAB_FE_CERTS/robotcert.pem
    cp ~/.globus/userkey.pem  $CRAB_FE_CERTS/robotkey.pem
  • set LANDB alias
    openstack server set --property landb-alias=$CMSWEB_CLUSTER $CRAB_FE_MINION
  • point browser to https://ca.cern.ch/ca/host/HostSelection.aspx?template=CERNHostCertificate2YearsCustomSubject&instructions=auto and obtain host certificate for crabdev-frontend.cern.ch It must appear with that name in the pulldown menu (may need to reload page).
  • do not set any password for the certificate, download it ( it is named crabdev-frontend.p12 i.e. ${CMSWEB_CLUSTER}.p12 ) and place it in $CRAB_FE_CERTS
  • generate pem files
    pushd $CRAB_FE_CERTS
    openssl pkcs12 -clcerts -nokeys -in ${CMSWEB_CLUSTER}.p12 -out  cmsweb-hostcert.pem
    openssl pkcs12 -nocerts -nodes -in ${CMSWEB_CLUSTER}.p12 -out  cmsweb-hostkey.pem
    chmod 400 *.pem


User OpenStack to create another cluster, now node can be medium, call it crabdev-backend Repeat the above with trivial changes
export CMSWEB_CLUSTER=crabdev-backend
export CRAB_BE_DIR=/afs/cern.ch/user/b/belforte/WORK/K8s/BE
mkdir -p ${CRAB_BE_DIR}
openstack coe cluster config  crabdev-backend
export KUBECONFIG=${CRAB_BE_DIR}/config
kubectl get nodes
CRAB_BE_MASTER=`kubectl get node|grep master|awk '{print $1}'`
CRAB_BE_MINION=`kubectl get node|grep minion|awk '{print $1}'` 
mkdir -p $CRAB_BE_CERTS
cp ~/.globus/usercert.pem  $CRAB_BE_CERTS/robotcert.pem
cp ~/.globus/userkey.pem  $CRAB_BE_CERTS/robotkey.pem
openstack server set --property landb-alias=$CMSWEB_CLUSTER $CRAB_BE_MINION
get cert from ca.cern.ch, dwnload it and place it in $CRAB_BE_CERTS as above, then
openssl pkcs12 -clcerts -nokeys -in ${CMSWEB_CLUSTER}.p12 -out  cmsweb-hostcert.pem
openssl pkcs12 -nocerts -nodes -in ${CMSWEB_CLUSTER}.p12 -out  cmsweb-hostkey.pem
chmod 400 *.pem
Deployment is different
pushd $CMSWEB_K8s/kubernetes/cmsweb-nginx
export CMSWEB_SERVICES="crabserver crabcache"
export CMSWEB_HOSTNAME=crabdev-frontend.cern.ch
./scripts/gen_hmac.sh hmac
./scripts/deploy.sh create services $CMSWEB_CONFIG $CRAB_BE_CERTS hmac

-- StefanoBelforte - 2019-10-29

Edit | Attach | Watch | Print version | History: r2 < r1 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r2 - 2019-10-29 - StefanoBelforte
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback