Timepix3 Telescope Setup

Telescope server

After restart of the server, do once:

• sudo modprobe ni_usb_gpib (if necessary)
• sudo gpib_config
• sudo chmod a+rw /dev/gpib0
• For oscilloscope readout (e.g. FastPix): "sudo ip addr add 192.168.5.1/24 dev eth3". Then run "remmina" and connect to "Oscilloscope".
• check if IP address of server has changed (it is used as parameter for the Peary producers).
• Then start the procedure below.

In case of problem with the nfs mounts of the server /data directory on other machines like the Zynq boards (not able to write to them anymore), the nfs server needs to be restarted:

• sudo service nfs-kernel-server restart

If no module "ni_usb_gpib" is available for the current kernel version, it has to be recompiled:

• Go to "Downloads/linux-gpib-4.1.0" (or download the latest linux-gpib package)
• ./configure
• make
• sudo make install
• Check that your new kernel has the modulesby checking if this directory exists:
• ls -l /lib/modules//gpib
• Then continue with above steps of loading the module.

To move the telescope box using the motion stages:

• cd ~/telescope-motion
• python motion.py (may have to run this several times, before the actual GUI appears)
• Gitlab repository, including more documentation: https://gitlab.cern.ch/nurnberg/telescope-motion

Start the telescope DAQ:

• from pclcd47: ssh -Y telescope@clictelescope
• To start processes for the TLU and telescope DACs
  • cd SPIDR/software/Telescope/Scripts
  • ./startTelescope.sh
• To run the run control Gui:
  • cd SPIDR/software/Telescope/RunControl/RCGui
  • make (if any changes made)
  • python hv_server.py (for high voltage) or dummy_hv_server.py (in case HV is controlled by hand, for debugging etc.)
  • python serial_server.py (for motion stage, not used currently!)
  • then in a different terminal window: ./RCGui
  • click on "connect HV server".
• NEW 2022: You can also automatically start all processes on the server by running the scrip "restartTelescope.sh" in the home directory. (Remember that you still have to start the pearysrv process on the Zynq machine.)

MCP time reference (for FASTPIX data taking)

• Make sure MCP High Voltage is set correctly (2600-3300 V). Current ~74 uA @ 2600 V.
• Connect / control remotely from server:
  • for 4-channel version (CAEN N470): 'screen /dev/ttyACM0 9600',
  • for 2-channel version (CAEN N1471A): 'screen /det/ttyUSB13' (May 2022, check with 'lsusb'...)
  • type 'caen' and press return. 'ctrl+a, ctrl+d' to end screen session.

FASTPIX data taking:

• open terminal on Zynq computer (from pclcd47): ssh root@pclcd-lab-zynq (may need to replace name of machine with the actual name set via the SD card)
• run "./ip.sh" to be able to connect to oscilloscope (only if machine was rebooted)
• check that you can still write to /data: "touch /data/test.txt". If error message: restart nfs server on clictelescope (see above).
• check link speed: "dmesg". There should be a line like "[ 7.959350] macb e000b000.ethernet eth0: link up (1000/Full)". In case the last part does not say 1000/Full, but something smaller, reset the link speed to 1 Gbit: "ethtool -s eth0 speed 1000 duplex full"
• from a web browser on pclcd47, connect to the Keithley HV for FASTPIX: lcdkeith3.cern.ch. Select "Virtual front panel". User name admin, and the usual password. Set bias to -6V, 105uA current limit. (Only needed when starting from scratch.)
• Check fastpix.cfg for the correct threshold setting. Can tune threshold by using "pearycli -c fastpix.cfg FASTPIX" and checking the scope output for noise depending on which matrix and threshold are selected.
• Start FASTPIX producer: "pearysrv -c fastpix.cfg -i 128.141.75.66 -d /data/ -v INFO". The IP address is the one of the clictelescope server. It may change from time to time, as it is not configured for fixed IP. Check that the bias does not trip (all power supplies should be at "1", and the Keithley should show the correct bias and a few uA current). If it trips, restart the producer until it is fine.
• Select the matrix index in the RCGui by entering it in the field "Default thr_fine". (This is normally used to select the threshold. But for FASTPIX we use it for the matrix selection, to avoid adding another parameter.)
• For automatic runs with default file-size limit (normally 550MB), enter a large number in the field "Runs/measurement" and "Add scan parameters to measurement list".
• To start runs: "Start measurement list". Monitor progress in "Display monitoring" tab.
• To end runs: "End measurement list".

Region-Of-Interest (ROI) mask

• ...

Timepix3 data taking with Spidrman for finding beam spot etc.

• cd SPIDR/software/python/ui
• ./spidrman.py
• Menu: SPIDR -> settings. Select the IP address / port of the SPIDR interface to be displayed (default connects to second upstream plane W13_E3).
• Menu: File -> Connect -> Spidr. File -> Load Configuration. Select last configuration of the connected device.
• Enable relevant tabs in "View", in case they are not shown.
• Checke Threshold and change as appropriate (for example, should be 1230 for W2_J5)
• There is a bug in the polarity setting (electrons/holes are reversed), and the polarity is not stored correctly in the configuration file. So for the p-in-n telescope sensors the polarity should be set to electrons and one should check / adapt the general config setting in the t3x configuration file.
• [For default plane, can try with DAQ "SPIDR" in SPIDR -> Settings, then load config "W13_E3_equalised.t3x" and set threshold to about 1175. The baseline should be approximately 1045 for this sensor.]
• Select for example ToT or ToA tab. Open shutter by clicking on "On". Change decay in visualisation tab to 99. Look at rate.

SPIDR board communication

The IP of each SPIDR board is determined by the chipboard plugged in to FMC connection 0 (the FMC connector closest to the 10 Gbit ethernet). The default IP is 192.168.100.10. To allow several SPIDR to be connected to the same computer, each of the boards should be assigned a separate IP. This assignment of IPs to chips is:

Chip ID	IP
W13 E3	192.168.100.10
W13 G2	192.168.101.10
W13 G3	192.168.102.10
W13 J5	192.168.103.10

The ethernet ports by default are set to the IP address 192.168.10X.1, for connection ethX. Therefore in principle the SPIDR with chipboard W13 E3 should be connected to eth0. If this is not the case, then the network manager on the linux machine must be used to manually change the IP of the port (ipconfig will be overridden - do not use it!).

The IPs of the chip boards can be re-configured using the tool spidripconfig in ~/SPIDR/software/Release/. For example:

./spidripconfig 192.168.101.10 192.168.100.10 dest 192.168.100.1

changes the IP address from 192.168.101.10 to the default 192.168.100.10, with connection to the interface assigned to IP 192.168.100.1.

SPIDR board reconfiguration and monitoring

The SPIDR boards can be reprogrammed and monitored remotely. The JTAG server listens on the default port 3121. The JTag chains monitoring UART mapping is the following:

Name of the JTag chain	UART device	Interface assigned to the given SPIDR card	IP
Digilent/210203A0236FA	/dev/LeonSerial102	eth0	192.168.102.10
Digilent/210203A023ADA	/dev/LeonSerial103	eth5	192.168.103.10
Digilent/210203A02F16A	/dev/LeonSerial101	eth11	192.168.101.10
Digilent/210203A02EF0A	/dev/LeonSerial100	eth7	192.168.100.10

-- AdrianFiergolski - 2016-04-28

To reboot SPIDR board, run on clictelescope server:

• source /tools/Xilinx/Vivado_Lab/2022.1/settings64.sh
• vivado_lab
• select 'open hardware manager'
• open target -> auto connect, the 4 devices should show up in the list
• if not yet open, right-click on device and select 'open device'
• right click on fpga (xc7vx485t) and select 'boot from configuration memory device'

Telescope planes order in z:

Upstream

• W0013_D04
• W0013_E03
• W0013_G02

DUT

• W0013_G03
• W0013_J05
• W0013_L09

Downstream

</verbatim>

SPIDR instructions

• https://docs.google.com/document/d/1OCOTdJOl51ziii-_Oc6WTDVCB0zcaWRhVHKMINIMAT0/edit#heading=h.mix7jqx6l2r

TLU

• TLU busy masks:
  • 0x40 : SPIDR 103
  • 0x80 : SPIDR 102
  • 0x100: DUT2
  • 0x200: DUT1
  • 0x400: SPIDR 101
  • 0x800: SPIDR 100

InvestigatorDataTaking

HDMI to RJ45 cable (trigger system)

How to change to new run directory for new run period

• Example: Preparing for September 2021 beam period:
  • new directory /data/tbSeptember2021, /data/tbSeptember2021/data, /data/tbSeptember2021/log, /data/tbSeptember2021/fastpix
  • copy directory /data/tbAugust2021/config to /data/tbSeptember2021/ (incl. subdirectories, using "cp -r").
  • changing DAQFilePath to /data/tbSeptember2021/config and logpath to /data/tbSeptember2021/log in /home/telescope/SPIDR/software/Telescope/RunControl/RCGui/config/test.cfg
  • Changing in ./home/telescope/SPIDR/software/Telescope/DAQ/Tpx3daq.cpp: #define CFG_PATH "/data/tbAugust2021/config" #define DATA_PATH "/data/tbAugust2021/data" #define LOG_PATH "/data/tbSeptember2021/log"; run "make" in this directory to recompile
  • Change mount directory in /etc/fstab on Zynq board: "clictelescope:/data/tbSeptember2021/fastpix /data nfs defaults 0 0" and in /etc/exports on clictelescope: "/data/tbSeptember2021/fastpix pclcd-lab-zynq(rw,async,no_subtree_check,all_squash,anonuid=1000,anongid=1000)". restart nfs server on clictelescope: "sudo service nfs-kernel-server restart"

Direct remote access to data on telescope server via ssh mount (works on lxplus)

• create local directory as mount point, e.g. "mkdir ~/tbdata".
• sshfs -o allow_other,default_permissions telescope@clictelescope:/data ~/tbdata
• can now access telescope via the director ~/tbdata.

Copy data to EOS (updated 08/2021)

• on clictelescope, make CVMFS available and then start the Docker image:
  • place your own personal grid certificate (userkey.pem, usercert.pem) in the ~/.globus folder (see below for getting a certificate and registration in ilcdirac [1])
  • ls /cvmfs/clicdp.cern.ch/DIRAC
  • sudo docker run -i --rm -t -v ~/.globus:/root/.globus -v /:/root/local_machine -v /cvmfs:/cvmfs -v /var/run/nscd:/var/run/nscd --net=host gitlab-registry.cern.ch/clicdp/ilcdirac/ilcdirac-ci-images:cc7 /bin/bash
• In the the container do:
  • cd; source /cvmfs/clicdp.cern.ch/DIRAC/bashrc
  • In case you get the error "bash: /cvmfs/clicdp.cern.ch/DIRAC/bashrc: Too many levels of symbolic links", exit the container and kill the dockerd process: sudo killall dockerd. Then try again.
  • dirac-proxy-init -g clic_silicon
  • the password is that of your own certificate
• Make sure the destination folder in the filecatalog exists - if not, create it like this:
  • dirac-dms-filecatalog-cli
  • cd /ilc/prod/clic/clic_silicon/testbeam
  • mkdir CERN_TB_Month_20XX
  • exit
• Use the following command to synchronize the local test beam directory to the grid space:
  • dirac-dms-directory-sync -j 8 local_machine/__PATH_TO_TB_DATA__ /ilc/prod/clic/clic_silicon/testbeam/CERN_TB_Month_20XX CERN-DST-EOS
• You have to prepend local_machine before the path since this is the mount point.
• dirac-dms-directory-sync can be parallelized by using the flag -j same as in make.
• Files cannot be overwritten. If a file changed since the last sync, it has to be removed manually, otherwise the synchronization will not complete. This happens mostly to the run logfile, e.g.:
  • dirac-dms-remove-files /ilc/prod/clic/clic_silicon/testbeam/CERN_TB_Month_20XX/log/global_log.txt
• The Docker container can be run in the background by simply detaching form it with CTRL+P CTRL+Q. To reattach, get the list of running containers via sudo docker ps and then run sudo docker attach

• Data will be accessible from lxplus as: /eos/experiment/clicdp/grid/ilc/prod/clic/clic_silicon/testbeam/CERN_TB_Month_20XX
• Data from 2018 and before can be found in /ilc/user/c/clictel/ (/eos/experiment/clicdp/grid/ilc/user/c/clictel/ from lxplus)

[1] https://lcd-data.web.cern.ch/lcd-data/doc/ilcdiracdoc/DOC/Files/UserGuide/registration.html

Verify Copied Data

• EOS does not support has sums currently, so simply check for the files and their size, e.g. run the following command on the telescope server and on EOS (e.g. via lxplus)
• find . -type f -printf "%p %s\n" | sort > ~/files_on_EOS.txt
• find . -type f -printf "%p %s\n" | sort > ~/files_on_TEL.txt
• diff <(sort files_on_TEL.txt) <(sort files_on_EOS.txt)
• If there's no difference (or only zero-byte log files, which are not copied to EOS) everything is in order.