Instructions for working on ndpc3 remotely

* Note that you will have to contact the sysadmins in order to set up account on ndpc3 if you do not currently have one *

VNC

The following steps detail how to establish a GUI session on ndpc3, assuming you are using a Linux computer

1: VNC setup

The following is done on ndpc3. In principal this needs to be done only a single time in order to set your account up for vnc.

1.1: Create a VNC password

vncpasswd

When prompted, enter (and then confirm) a new password.

1.2: Generate an xstartup file for vnc

echo '#!/bin/sh' > ~/.vnc/xstartup && echo 'exec gnome-session' >> ~/.vnc/xstartup

chmod 755 ~/.vnc/xstartup

1.3: Start the VNC server

vncserver :<vnc_session_id> -geometry <resolution>

<vnc_session_id> should be picked to be a number which is not currently in use. Check with sysadmins to determine a suitable number.

<resolution> is the resolution of the session. e.g. 1920x1080

1.4: Verify that the VNC server is running

vncserver -list

You should see something like

X DISPLAY # PROCESS ID
:<vnc_session_id> <pid>

2: Open a tunnel

The following should be run from your personal computer. This needs to be done every time you would like to establish a VNC session.

2.1: First Tunnel

ssh -L 2222:ndpc3:22 <nice_user>@lxplus.cern.ch

<nice_user> is your standard CERN login. When prompted for your password, enter your NICE password.

2.2: Second Tunnel

ssh -C -p 2222 -L <port>:localhost:<port> <ndpc3_user>@localhost

<port> should be 5900 + <vnc_session_id>. e.g. if <vnc_session_id> is 3 then <port> is 5903.

<ndpc3_user> is the name of your account on ndpc3. When prompted for a password, enter the password for your ndpc3 account. Exceptional circumstances notwithstanding, your ndpc3 account should (username and password) should be the same as your NICE account.

3: VNC client setup

3.1: Obtain VNC client software

This page will be using TigerVNC as an example. TigerVNC is open source and cross platform.

For those on debian based systems:

sudo apt-get install tigervnc-viewer

3.2: Connect to ndpc3

Launch your VNC client.

To launch TigerVNC on linux:

vncviewer

When prompted for the server, enter:

localhost:3

When prompted for the password, enter the password which you created with the vncpasswd command in step 1.1.

You should now see the desktop for ndpc3.

Vivado

1: Connect to ndpc3 via VNC

Use the previous instructions to establish a VNC connection to ndpc3.

Once on the ndpc3 desktop, launch a terminal.

2: Configure the environment variables

In the terminal which you have launched, run the following command:

source /tools/Xilinx/Vivado/2019.1/settings64.sh

3: Launch Vivado

In the same terminal, run the command

/tools/Xilinx/Vivado/2019.1/bin/vivado &

This should launch Vivado's GUI.

4: Make the changes permanent

If the previous steps were successful, in the terminal run

touch ~/.bashrc && echo 'source /tools/Xilinx/Vivado/2019.1/settings64.sh' >> ~/.bashrc && echo 'PATH=$PATH:/tools/Xilinx/Vivado/2019.1/bin' >> ~/.bashrc && echo 'export XILINXD_LICENSE_FILE=2112@lxlicen01,2112@lxlicen02,2112@lxlicen03' >> ~/.bashrc

Note that we also export the licenses here which you will need to build some projects.

You should now be able to launch vivado by running the command

vivado

from the terminal (you will probably have to log out and back in for the changes to take effect).

Set up SSH keys

1: Create a local ssh key

On ndpc3 run

ssh-keygen -t rsa -b 2048 -C

This will create a public private key pair associated to your ndpc3 account.

2: Copy the public key

Copy the contents of

~/.ssh/id_rsa

How you copy this will depend on which terminal you are using.

3: Paste the keys into gitlab / github

Here I will use gitlab as an example, although the procedure should be nearly the same for github.

3.1: Log into gitlab

Log into gitlab.cern.ch using your NICE account.

3.2: Navigate to SSH keys management

User Settings > SSH keys

3.3: Add the public key

Paste the contents of ~/.ssh/id_rsa into the 'Key' text box. Title the key 'ndpc3' and enter a reasonable expiration date. Then click 'Add Key'.

Build the BCPv1 firmware

0: Prerequisites

Note that before doing the following you will need

  1. ssh keys set up for gitlab.cern.ch (see 'Setting up SSH keys above')
  2. ssh keys set up for github.com (see 'Setting up SSH keys above')
  3. permissions to access the project on gitlab.cern.ch
  4. Vivado and license files sourced. Doing this in .bashrc is easiest (see 'Vivado' above)
  5. an open VNC connection to open the Vivado gui in the final step (see 'VNC' above)
    • Note that in principle you could build the project in a ssh session and then perform only the final step in the VNC session

1: Clone the project

On ndpc3 run

git clone --recursive ssh://git@gitlab.cern.ch:7999/cms-bcp/bcpd/fpga/bcpd_ibert.git

Note that you may need to be given permissions on the gitlab project before you are able to do this.

Note also that this uses the --recursive option to clone the ruckus project which lives on github (not gitlab) and thus you will need ssh keys set up on both gitlab.cern.ch and github.com to run this command successfully.

2: cd into the new project

Run

cd bcpd_ibert

3: Create a build directory

Run

mkdir build

4: cd to the firmware directory

Run

cd firmware

5: Build the project

Run

make

The build will take ~ 1 hour. If the build is successful you will see something like

# SourceTclFile ${VIVADO_DIR}/post_build.tcl
# close_project
# exit 0
INFO: [Common 17-206] Exiting Vivado at Tue Oct 13 11:03:57 2020...

6: Launch the gui

Run

make gui

The project should open in a Vivado gui window (thus this needs to be done in a vnc session).

Power cycle the BCP

0: Try to ping the ELM

0.0: Determine the elm address

In order to get a list of aliases, check the hosts file:

less /etc/hosts

or to print just the ELMs:

grep elm /etc/hosts

As of Nov. 2020 we have the following (although this is subject to change):

BCP number ELM IP ELM alias IMPC IP IPMC ALIAS purpose
32 192.168.40.201 elm10032 192.168.40.200 ipmc10032 Trigger integration testing
22 192.168.40.203 elm10022 192.168.40.202 ipmc10022 ECAL developement
11 192.168.40.205 elm10011 192.168.40.204 ipmc10011 HCAL integration testing

0.1: Try to ping the ELM

On ndpc3 run

ping -c 3 <elm>

Where <elm> is the IP address of the ELM, or it's local alias.

A 'healthy' ELM will be pingable. Example:

ping -c 3 elm10011
PING elm10011 (192.168.40.205) 56(84) bytes of data.
64 bytes from elm10011 (192.168.40.205): icmp_seq=1 ttl=64 time=0.189 ms
64 bytes from elm10011 (192.168.40.205): icmp_seq=2 ttl=64 time=0.192 ms
64 bytes from elm10011 (192.168.40.205): icmp_seq=3 ttl=64 time=0.174 ms

--- elm10011 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.174/0.185/0.192/0.007 ms

If an ELM is not pingable (example):

ping -c 3 elm10011
PING elm10011 (192.168.40.205) 56(84) bytes of data.
From dnsmasq (192.168.40.254) icmp_seq=1 Destination Host Unreachable
From dnsmasq (192.168.40.254) icmp_seq=2 Destination Host Unreachable
From dnsmasq (192.168.40.254) icmp_seq=3 Destination Host Unreachable

--- elm10011 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 1999ms
pipe 2

then the BCP likely need to be power cycled (see the next step).

1: Power cycle the BCP

1.0: Locate the IPMC address

We need to find the address of the IPMC on out BCP. We can again check the hosts file and loop for IPCs:

grep ipmc /etc/hosts

As of Nov. 2020 the table in the section 0.0 is accurate.

1.1: Telnet to the IMPC

On ndpc3 run:

telnet <ipmc>

Where <ipmc> is the IP address of the IPMC, or it's local alias.

You will be prompted for a password. If you do not already know it, contact a sysadmin to get the password.

1.2: Issue the power cycle commands

From the IPMC run

payload.power_level 0 force

payload.power_level 1 force

then return to ndpc3. Verify that the ELM is pingable.

Note: there may be a delay of up to 30 seconds between the power cycle and the elm becoming pingable.

2. Start the Xilinx Virtual Cable

2.1: ssh into the ELM

From ndpc3:

ssh root@<elm>

where as before <elm> is the IP address of the ELM, or it's local alias.

You will be prompted for a password. If you do not already know it, contact a sysadmin to get the password.

2.2: Check that XVC is not already running

From the ELM, run

ps a | grep xvc

If you see only your grep

ps a | grep xvc
1390 root 0:00 grep xvc

this means that no VNC process is running.

2.3: Start XVC

On the ELM, run

xvc <ndpc3_ip> &

where <ndpc3_ip> is the IP adress of ndpc3 on the local network.

As of Nov. 2020 it is 192.168.40.254 . This is, in principle subject to change. In order to verify the ndpc3 IP you can run on ndpc3

ifconfig | grep -A1 enp8s0

You will see something like

ifconfig | grep -A1 enp8s0
enp8s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.40.254 netmask 255.255.255.0 broadcast 192.168.40.255

where the inet address (bolded here) tells you ndpc3's local IP.

Verify that XVC is running:

ps a | grep xvc

You should now see and xvc process in addition to your grep:

ps a | grep xvc
1395 root 0:00 xvc 192.168.40.254
1397 root 0:00 grep xvc

Exit the ELM.

2.4: Check XVC in Vivado

On npdc3 launch Vivado.

Under 'Tasks' select 'Open Hardware Manager'.

In the 'Harware' panel select the 'Autoconnect' button.

If you see a connected 'xcku115_0' FPGA you are good to go. If not you may have to manually add the XVC. To do this, right click 'localhost' in the 'Hardware' panel. Then select 'Add Xilinx Virtual Cable'. When prompted, for 'Host Name' use the IP of the ELM. Note that you must use the IP address and not the alias here. For 'Port' use the XVC default (2542). Then hit 'Ok'. You should now see the attached 'xcku115_0'.

3: Program the FPGA

Open the Vivado Hardware Manager and verify that the FPGA is connected using the steps in part 2.4.

Now right click on 'xcku115_0' and select 'Program Device'. When prompted, provide the path to the programming bitfile and then click 'Program'. If you see ... then the programming was successful.

4: Configure and enable the clocks

Note: This could also be done before programming the FPGA.

4.0: Determine the Si5345 addresses

Each Si5345 has both an I2C address and an ELM address. The following values are current as of Nov. 2020.

Si5345 Name Si5345 Label on BCP silk I2C address ELM address purpose
ASYNC IC623 0x68 0x41270000 Provide asynchronous refclks to GTH transceivers
SRC IC622 0x69 0x412b0000 Switch between internal and external clk for SYNC
SYNC IC626 0x6a 0x412d0000 Provide synchronous refclks to GTH transceivers

4.1: ssh into the ELM

From ndpc3:

ssh root@<elm>

where as before <elm> is the IP address of the ELM, or it's local alias.

You will be prompted for a password. If you do not already know it, contact a sysadmin to get the password.

4.2: Configure the Si5345's

For each Si5345 follow the following procedure (on the ELM).

Load the cfg file:

si534x <ELM_I2C_device> <Si_I2C_address> -v -c </path/tp/cfg_file>

Here <ELM_I2C_device> is the linux device corresponding to the I2C bus. As of Nov. 2020 this is /dev/i2c-3.

<Si_I2C_address> is the I2C address from the table in section 4.0.

</path/tp/cfg_file> is the path to the configuration file to be loaded. Note that in general each Si5345 requires a unique cfg file. The are usually stored in the SD card at /mnt/persistent/bcp_user.

4.3: Enable the Si5345 outputs:

For each Si5345 follow the following procedure (on the ELM).

poke <ELM_address> 0x2

where <ELM_address> can be found in the able in section 4.0.

The check that the write was successful:

peek <ELM_address>

You should read back 0x2. Example:

peek 0x41270000
0x00000002

Note: If you want to disable a Si5345, write 0x3:

poke <ELM_address> 0x3

4.4: Check the Si5345 status

Still on the ELM, run:

si534x <ELM_I2C_device> <Si_I2C_address> -v -s -i

Here <ELM_I2C_device> is the linux device corresponding to the I2C bus. As of Nov. 2020 this is /dev/i2c-3.

<Si_I2C_address> is the I2C address from the table in section 4.0.

Generate Si5345 configuration files

Note the the follow requires use of the Si ClockBuilder Pro software which runs only on windows. Thus access to a Windows computer on which one can install software is neccessary for the foillowing procedure.

Connect to ndscope1 remotely

ndscope1 is running an rdp server. You can connect to it using an rdp client.

Note that unlike VNC, rdp grabs the primary display, so the the physical scope will be locked when someone is connected via rdp and only one rdp session is possible at a time.

Windows

Windows has a built in rdp client. You can access it by searching for 'Remote Desktop Connection'.

windows_ndscope1_1.png


When prompted for the 'Computer' enter ndscope1.cern.ch.

windows_ndscope1_2.png

You will then be prompted for a username and a password. You will need to get these credentials from the admins. Note that if you connect from within the CERN network, the username will default to your NICE account and you will have to select 'More choices' >> 'Use a different account' in order to log in with the local scope account.

windows_ndscope1_3.png

Linux

You will need to install an rdp client for Linux. I will use remmina for this example.

Set up a profile with the following settings:

remmina_ndscope1_1.png

You can then connect with this profile by double-clicking it in the remmina menu. You will then be prompted for a username and a password. You will need to get these credentials from the admins.

remmina_ndscope1_2.png

-- JosephPatrickMariano - 2020-10-03

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng remmina_ndscope1_1.png r1 manage 40.3 K 2021-03-17 - 15:56 JosephPatrickMariano  
PNGpng remmina_ndscope1_2.png r1 manage 19.7 K 2021-03-17 - 15:56 JosephPatrickMariano  
PNGpng windows_ndscope1_1.png r1 manage 11.3 K 2021-03-17 - 15:57 JosephPatrickMariano  
PNGpng windows_ndscope1_2.png r1 manage 8.4 K 2021-03-17 - 15:57 JosephPatrickMariano  
PNGpng windows_ndscope1_3.png r1 manage 6.4 K 2021-03-17 - 15:57 JosephPatrickMariano  
Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r8 - 2021-03-17 - JosephPatrickMariano
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback