The dune_artdaq repository (among others) is hosted on the fermilab git repository, so changes need to be pushed there. But np04-srv-* machines are behind a firewall, so external traffic has to go through the web gateways or through ssh tunnels on lxplus. There's an additional complication that both CERN and FNAL use kerberos for authentication, so you need both CERN and FNAL tickets to be active in order to tunnel through lxplus to the fermilab git repos.

All of the following steps are done on the np04-srv-* machines and only need to be done once, unless otherwise stated.

The first step is to set up a modified kerberos configuration. Save the following as krb5.conf.fnaltunnel (say) somewhere that's accessible on the np04-srv-* machines:

[libdefaults]
default_realm = CERN.CH
ticket_lifetime = 25h
renew_lifetime = 120h
forwardable = true
proxiable = true
default_tkt_enctypes = arcfour-hmac-md5 aes256-cts aes128-cts des3-cbc-sha1 des-cbc-md5 des-cbc-crc
# PAR 2018-03-12: Use the "DIR" type of credentials cache so we can have multiple credentials
# See:
# https://web.mit.edu/kerberos/krb5-1.12/doc/basic/ccache_def.html
default_ccache_name = DIR:/tmp/krb5cc_%{uid}

[realms]
# PAR 2018-03-12: Copied from /etc/krb5.conf since we don't need any proxying funny business for CERN
 CERN.CH = {
  default_domain = cern.ch
  kpasswd_server = cerndc.cern.ch
  admin_server = cerndc.cern.ch
  kdc = cerndc.cern.ch

  v4_name_convert = {
     host = {
         rcmd = host
     }
  }
 }

# Tunnel fnal kerberos through port 8899. Needs a proxy on that port, eg via:
# ssh -L 8899:krb-fnal-1.fnal.gov:88 -N lxplus
FNAL.GOV = {
 kdc = localhost:8899
 admin_server = krb-fnal-admin.fnal.gov
 default_domain = fnal.gov
}

[domain_realm]
.cern.ch = CERN.CH
.fnal.gov = FNAL.GOV

[kadmin]
default_keys = arcfour-hmac-md5:pw-salt

(modified from krb5.conf.tunnel at here)

Make sure kerberos commands see this config file by setting $KRB5_CONFIG (eg export KRB5_CONFIG=/path/to/krb5.conf.fnaltunnel in your ~/.bashrc). Also make sure the environment variable $KRB5CCNAME is not set (eg unset KRB5CCNAME in your ~/.bashrc), since it will override the setting in krb5.conf.fnaltunnel.

krb5.conf.fnaltunnel tells kerberos commands to try to connect to the FNAL kerberos servers via localhost:8899 so we need an ssh tunnel to forward connections to localhost:8899 to the actual FNAL kerberos servers. For example, run:

 ssh -L 8899:krb-fnal-1.fnal.gov:88 -N your_cern_username@lxplus.cern.ch

This step has to be done each time you log in (and want to use FNAL machines).

Now make sure you have both CERN and FNAL kerberos tickets using kinit (replace cern_principal and fnal_principal with your actual usernames):

kinit cern_principal@CERN.CH
kinit fnal_principal@FNAL.GOV

You can check that both are active with klist -l which should show something like:

Principal name                 Cache name
--------------                 ----------
fnal_principal@FNAL.GOV              DIR::/tmp/krb5cc_123456/tktrsk9rM
cern_principal@CERN.CH               DIR::/tmp/krb5cc_123456/tkt

The last step is to tell ssh to tunnel through lxplus to get to any FNAL machines. If you already have an ~/.ssh/config that works for FNAL access when there's no firewall, you just need to add sections like this to ~/.ssh/config (with the appropriate replacements):

Host *.fnal.gov
  PreferredAuthentications gssapi-with-mic,gssapi,password
  ProxyJump cern_principal@lxplus.cern.ch   # Replace with your actual CERN username
  User fnal_principal # Replace with your actual FNAL username

Host cdcvs.fnal.gov
  PreferredAuthentications gssapi-with-mic,gssapi,password
  ProxyJump cern_principal@lxplus.cern.ch  # Replace with your actual CERN username

If you don't have an ~/.ssh/config already, a complete example (again modified from here) looks like:

Host *.cern.ch *.fnal.gov
  GSSAPIAuthentication yes
  GSSAPIDelegateCredentials yes
  PubkeyAuthentication yes

Host *.fnal.gov
  PreferredAuthentications gssapi-with-mic,gssapi,password
  ProxyJump cern_principal@lxplus.cern.ch
  User fnal_principal

Host cdcvs.fnal.gov
  PreferredAuthentications gssapi-with-mic,gssapi,password
  ProxyJump cern_principal@lxplus.cern.ch

Host *.cern.ch
  GSSAPITrustDNS no

Host *
  IdentitiesOnly yes
  ServerAliveInterval 5
  ServerAliveCountMax 6
  GSSAPIRenewalForcesRekey yes
  GSSAPITrustDNS yes
  ForwardX11 no
  Protocol 2

Now that you have kerberos tickets for FNAL and CERN set up, along with the configuration to tunnel ssh connections through lxplus, you should be able to log in to any FNAL machine that you have access to (eg, the dune gpvms). If you can do that, then everything is set up correctly.

One last thing for pushing to git repositories is that the remote URL must be set correctly. git remote -v will show you the remote URL(s) for your repository. It should look like ssh://p-PROJECTNAME@cdcvs.fnal.gov/cvs/projects/PROJECTNAME (replace PROJECTNAME as appropriate). If not, you can change it with git remote set-url origin ssh://p-PROJECTNAME@cdcvs.fnal.gov/cvs/projects/PROJECTNAME . -- PhilipAndrewRodrigues - 2018-03-13