RadosSync

Author: dominique.cressatti@cern.ch / dominique.cressatti@gmail.com

ToDO

subsitute -cmin with variable and / or talk about setting it to none on 1st run

Purpose

To backup the reports files (and some others) generated on the RADOS units servers, using a set of RadoSync.pl Perl scripts and make those files accessible to users over the CERN /eos/project/h/hse-rp-cs/RadioLab/ Linux/Windows workspace. And while a solution based on rsync would have been preferred, due to the absence of rsync on the RADOS servers and the impossibility to install any type of synchronization software on them and given that it was necessary to prefix the report files with the backup date, and given that doing all would have rather complex to develop a solution with a shell script. I decided to code a solution in Perl.

Architecture

The architecture is composed of:

The rados units servers on which the report files are generated (in PDF and text format as well as other key files) in the rados units servers:/transfma directories which we need to backup. The rados units servers run on QNX which appears to be based on NetBSD. For more info see this User Guide.
The rados-server is a CERN OpenStack VM which runs on Linux CC7. It is “man in the middle”, which through a set of RadoSync.pl Perl scripts takes care of retrieving the report files from the rados units servers, copying the file locally along with prefixing each file with the date stamp of the backup date, before copying them again to the CERN /eos/project/h/hse-rp-cs/RadioLab/ workspace.
The CERN /eos/project/h/hse-rp-cs/RadioLab/ Linux/Windows workspace is where the report files are also backed up and are consulted from the Linux/Mac/Windows clients.

Thus we have 2 backup locations, 1st the rados-server and 2nd the /eos/project/h/hse-rp-cs/RadioLab/ workspace.

Detailed Mode of Operation:

As it was preferred to have a modular configuration instead of a monolithic one, there is a set of RadosSync.pl Perl scripts for each rados units servers , all residing in: rados-server:/home/rpgamma/<rados mobile unit dir>/script/. Each script is matched via a set of variables (see Variables section) to each rados units servers, and are running through a cron job (see Cron Job section below).

Each RadosSync.pl Perl scripts, poll every minutes over SSH, their corresponding rados units servers for new report files having been created in the last 5 days (hence why in the script, “-cmin” is set to 7200 minutes).
The new report files are then copied, using the SCP protocol to the rados-server into directories corresponding to their rados units servers (see below “RADOS Servers Directory Structure”), prefixing them along each file name with a date stamp in the “YYYY-MM-DD” format.
Finally the new report files (prefixed with a date stamp) are uploaded, using the WebDav protocol, to the CERN /eos/project/h/hse-rp-cs/RadioLab/ workspace, inside their respective <rados unit server>/transfma/ “target” base directories such as:
- SN40-Rados for rados-server:/home/rpgamma/SN40-Rados/transfma/ and /eos/project/h/hse-rp-cs/RadioLab/SN40-Rados/transfma/
- SN40-Rados for rados-server:rados-server:/home/rpgamma/SN134-Rados/ and /eos/project/h/hse-rp-cs/RadioLab/SN134-Rados/transfma/
- SN40-Rados for rados-server:/home/rpgamma/SN167-Rados/transfma/ and /eos/project/h/hse-rp-cs/RadioLab/SN167-Rados/transfma/

Cron Job:

There is a simple cron job owned by rpgamma (stored in /var/spool/cron/rpgamma) with an entry for each server script, having the following format:

* * * * * /usr/bin/perl /home/rpgamma/SN40-Rados/scripts/RadosSync_SN40.pl 2>&1 |tee /tmp/SN40_Sync_log
* * * * * /usr/bin/perl /home/rpgamma/SN134-Rados/scripts/RadosSync_SN134.pl 2>&1 |tee /tmp/SN134_Sync_log
* * * * * /usr/bin/perl /home/rpgamma/SN167-Rados/scritps/RadosSync_SN167pl 2>&1 |tee /tmp/SN167_Sync_log

Dependancies:

The RadoSync.pl Perl scripts running the Linux VM rados-server rely on the following PERL modules:

Net::OpenSSH
HTTP::DAV
Crypt::SSLeay

To install them, the following has been performed as root on the rados-server:

yum install perl-cpan
cpan install Net::OpenSSH
cpan install HTTP::DAV
cpan install Crypt::SSLeay

User Accounts And SSH Private/Public Key Pairs:

As the RadosSync.pl scripts, relies on SSH and SCP for remote access to all of the rados units servers , the user account admin-rados had to have been created on each rados units servers , by performing as root on each: rados units servers

passwd admin-rados

(Unlike on Linux, as the rados units servers run on QNX (a UNIX version based NetBSD), the “passwd” command, both creates the user account and sets the password).

To perform the “password-less” copy using SCP of the report files, an SSH RSA key pair is used.

On the rados-server the public / private keys are stored in rados-server:/home/rpgamma/.ssh/.
And on each rados units servers the public key is stored in /home/admin-rados/.ssh/authorized_keys.

To copy the public key on each rados mobile unit server, the following has been performed (as the rpgamma user account) for e`ach rados units servers:

cat ~/.ssh/rados-mobil_rsa.pub | ssh admin-rados@<rados_mobile_unit_server> "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys"

To avoid having to specify where the private key is stored as parameter of the the SSH and SCP commands, the following SSH rados-server:/home/rpgamma/.ssh/config file has been created:

Host rados*
    ServerAliveInterval 15

    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes
    StrictHostKeyChecking no

    user admin-rados
    IdentityFile ~/.ssh/rados-mobil_rsa

Variables:

my $rado_mobile_server_name ◀️ ‒ – – The rados units server name from which we gonna copy the data. For example rados644.
my $rado_mobile_server_user ◀️ ‒ – – The user name used to log on the rados units server (usually “admin-rados”).
my $rado_mobile_server_transfma_dir ◀️ ‒ – – The base directory name where the source files are located (usually “/transfma”).
my $target_dir ◀️ ‒ – – The target path where we copy the data. For example rados-server:/home/rpgamma/SN40-Rados/transfma.
my $EOS_user ◀️ ‒ – – The user name used to log onto EOS (usually “rpgamma”).
my $EOS_user_password ◀️ ‒ – – The user’s password used to log onto EOS.
my $EOS_transfma_dir ◀️ ‒ – – The base EOS directory name where the reports are going to be backed up into (usually “/transfma”).
my $sleepTime ◀️ ‒ – – The time the script waits before polling again it’s rados units server (default 60 seconds).

Debugging Options:

As hinted from the cron job each script output their respective log files such as: /tmp/SN40_Sync_log, /tmp/SN134_Sync_log and /tmp/SN167_Sync_log, each outputting:

The rados units servers such as: rados644 matched with SN40-Rados, radosmobil matched with SN134-Rados and Rados661 matched with SN167-Rados.
The Target dirs such as: SN40-Rados, SN134-Rados and SN167-Rados
The Total amount of Files which important in comparison with the Total Files amount of files processed, Total of New Files, the amount of Copied and skipped Files (those that have already been copied). Should the figures not adding up, points to an issue that has to be investigated (see how to increasing the debugging level below).
The full source Path/FileName of the new file to be copied from the rados unit server.
Any newly discovered source subdirectory that doesn’t already exist on the target directories and is created on the rados-server and on the CERN /eos/project/h/hse-rp-cs/RadioLab/ Linux/Windows workspace.
The name of the new file prefixed with the date stamp.
The full target Path/FileName (prefixed with the date stamp) to be copied on the rados-server and on the CERN /eos/project/h/hse-rp-cs/RadioLab/ Linux/Windows workspace.
Whether or not the file doesn’t exist on on the rados-server and on the CERN /eos/project/h/hse-rp-cs/RadioLab/ Linux/Windows workspace and in such a case having copied it.
The total of Total Files Processed, new files, copied and skipped files.

To view the logs you can either type: tail -f /tmp/<logname> or less /tmp/<logname>.

Increasing the debugging level

To find out at which step a problem is occurring, you can un-comment (by removing the # in front of a line of code) any line ending with # for debuging and any line like ## print “sleep\n”; sleep 5; # for debuging, then save the script and run it manually (ex: /home/rpgamma/<target>/script/script_name> ).

The Linux VM rados-server

As aforementioned, the rados-server is the server on which the RadosSync.pl scripts runs and where the report files are copied first and as can be seen from the following Directory Structure, there is the same consistent directory structure for each rados units servers .

A script directory containing a RadosSync.pl script, matched to it’s corresponding rados units servers.
A transfma directory where the content of each rados units servers:/transfma/ directory are copied into.

Directory Structure

rados-server:/
└──────────── home/
              └─── rpgamma/
                   ├────── .ssh/
                   │       ├─── config
                   │       ├─── rados-mobil_rsa
                   │       └─── rados-mobil_rsa.pub
                   ├────── SN40-Rados/   ◀️ ‒ – – matched to server: rados644
                   │       ├─── script/
                   │       │    └──── RadosSync_SN40.pl
                   │       └─── transfma/
                   │            ├────── data file1
                   │            ├────── data file2
                   │            └────── and so on ...
                   ├────── SN134-Rados/   ◀️ ‒ – – matched to server: radosmobil
                   │       ├──── script/
                   │       │     └──── RadosSync_SN134.pl
                   │       └──── transfma/
                   │             ├────── data file1
                   │             ├────── data file2
                   │             └────── and so on ...
                   └────── SN167-Rados/   ◀️ ‒ – – matched to server: Rados661
                           ├──── script/
                           │     └──── RadosSync_SN167pl
                           └──── transfma/
                                 ├────── dir1
                                 │       └─── data file1
                                 ├────── data file1
                                 └────── and so on ...

The RADOS units servers:

The rados units servers are the servers on which the reports of analyzed radio-active material are created.

RADOS Units Servers Directory Structure

rados mobile unit server:/
└─────────────────────── home/
                         ├─── admin-rados/
                         │    └────── .ssh/
                         │            └─── authorized_keys
                         transfma/
                         ├────── dir1
                         │       └─── data file1
                         ├────── data file1
                         └────── and so on ...

RADOS EOS Directory Structure

The EOS Directory Structure into which the reports are finally backed up is consistently organized like the rados-server:/home/rpgamma/ Directory Structure.

/eos/project/h/hse-rp-cs/RadioLab/
└─────────────────────────────── rpgamma/
                                 ├────── SN40-Rados/
                                 │       ├─── script/
                                 │       │    └──── SN40_Sync.pl
                                 │       └─── transfma/
                                 │            ├────── data file1
                                 │            ├────── data file2
                                 │            └────── and so on ...
                                 ├────── SN134-Rados/
                                 │       ├──── script/
                                 │       │     └──── SN134_Sync.pl
                                 │       └──── transfma/
                                 │             ├────── data file1
                                 │             ├────── data file2
                                 │             └────── and so on ...
                                 └────── SN167-Rados/
                                         ├──── script/
                                         │     └──── SN167_Sync.pl
                                         └──── transfma/
                                               ├────── dir1
                                               │       └─── data file1
                                               ├────── data file1
                                               └────── and so on ...