RADOS < Sandbox

<div id="doc" class="markdown-body container-fluid"><h1 id="RadosSync">RadosSync</h1><p>Author: <a href="mailto:dominique.cressatti@cern.ch" target="_blank" rel="noopener">dominique.cressatti@cern.ch</a> / <a href="mailto:dominique.cressatti@gmail.com" target="_blank" rel="noopener">dominique.cressatti@gmail.com</a></p><hr><h1 id="ToDO">ToDO</h1><ul>
<li>subsitute -cmin  with variable and / or talk about setting it to none on 1st run</li>
</ul><hr><h4 id="Purpose">Purpose</h4><p>To backup the reports files (and some others) generated on the RADOS units servers, using a set of <a href="http://RadoSync.pl" target="_blank" rel="noopener">RadoSync.pl</a> Perl scripts and make those files accessible to users over the CERN <strong>/eos/project/h/hse-rp-cs/RadioLab/</strong> Linux/Windows workspace. And while a solution based on rsync would have been preferred, due to the absence of rsync on the RADOS servers and the impossibility to install any type of synchronization software on them and given that it was necessary to prefix the report files with the backup date, and given that doing all would have rather complex to develop a solution with a shell script. I decided to code a solution in Perl.</p><hr><h4 id="Architecture">Architecture</h4><p>The architecture is composed of:</p><ul>
<li>
<p>The <strong>rados units servers</strong> on which the report files are generated (in PDF and text format as well as other key files) in the <strong>rados units servers:/transfma</strong> directories which we need to backup. The <strong>rados units servers</strong> run on <a href="https://en.wikipedia.org/wiki/QNX" target="_blank" rel="noopener">QNX</a> which appears to be based on NetBSD. For more info see this <a href="http://www.qnx.com/developers/docs/6.5.0/index.jsp?topic=%2Fcom.qnx.doc.neutrino_user_guide%2Faccounts.html" target="_blank" rel="noopener">User Guide</a>.</p>
</li>
<li>
<p>The <strong>rados-server</strong> is a CERN OpenStack VM which runs on Linux CC7. It is “man in the middle”, which through a set of <a href="http://RadoSync.pl" target="_blank" rel="noopener">RadoSync.pl</a> Perl scripts takes care of retrieving the report files from the <strong>rados units servers</strong>, copying the file locally along with prefixing each file with the date stamp of the backup date, before copying them again to the CERN <strong>/eos/project/h/hse-rp-cs/RadioLab/</strong> workspace.</p>
</li>
<li>
<p>The CERN <strong>/eos/project/h/hse-rp-cs/RadioLab/</strong> Linux/Windows workspace is where the report files are also backed up and are consulted from the Linux/Mac/Windows clients.</p>
</li>
</ul><p>Thus we have 2 backup locations, 1st the <strong>rados-server</strong> and 2nd the <strong>/eos/project/h/hse-rp-cs/RadioLab/</strong> workspace.</p><hr><h4 id="Detailed-Mode-of-Operation">Detailed Mode of Operation:</h4><p>As it was preferred to have a modular configuration instead of a monolithic one, there is a set of <strong><a href="http://RadosSync.pl" target="_blank" rel="noopener">RadosSync.pl</a></strong> Perl scripts for each <strong>rados units servers</strong> , all residing in: <strong>rados-server:/home/rpgamma/&lt;rados mobile unit dir&gt;/script/</strong>. Each script is matched via a set of variables (see Variables section) to each <strong>rados units servers</strong>, and are running through a cron job (see Cron Job section below).</p><ol>
<li>
<p>Each <strong><a href="http://RadosSync.pl" target="_blank" rel="noopener">RadosSync.pl</a></strong> Perl scripts, poll every minutes over SSH, their corresponding <strong>rados units servers</strong>  for new report files having been created in the last 5 days (hence why in the script, “-cmin” is set to 7200 minutes).</p>
</li>
<li>
<p>The new report files are then copied, using the SCP protocol to the <strong>rados-server</strong> into directories corresponding to their <strong>rados units servers</strong> (see below “RADOS Servers Directory Structure”), prefixing them along each file name with a date stamp in the “YYYY-MM-DD” format.</p>
</li>
<li>
<p>Finally the new report files (prefixed with a date stamp) are uploaded, using the WebDav protocol, to the CERN <strong>/eos/project/h/hse-rp-cs/RadioLab/</strong> workspace, inside their respective <strong>&lt;rados unit server&gt;/transfma/</strong> “target” base directories such as:</p>
<ul>
<li><strong>SN40-Rados</strong> for rados-server:/home/rpgamma/SN40-Rados/transfma/ and /eos/project/h/hse-rp-cs/RadioLab/SN40-Rados/transfma/</li>
<li><strong>SN40-Rados</strong> for rados-server:rados-server:/home/rpgamma/SN134-Rados/ and /eos/project/h/hse-rp-cs/RadioLab/SN134-Rados/transfma/</li>
<li><strong>SN40-Rados</strong> for rados-server:/home/rpgamma/SN167-Rados/transfma/ and /eos/project/h/hse-rp-cs/RadioLab/SN167-Rados/transfma/</li>
</ul>
</li>
</ol><hr><h4 id="Cron-Job">Cron Job:</h4><p>There is a simple cron job owned by rpgamma (stored in /var/spool/cron/rpgamma) with an entry for each server script, having the following format:</p><pre><code>* * * * * /usr/bin/perl /home/rpgamma/SN40-Rados/scripts/RadosSync_SN40.pl 2&gt;&amp;1 |tee /tmp/SN40_Sync_log
* * * * * /usr/bin/perl /home/rpgamma/SN134-Rados/scripts/RadosSync_SN134.pl 2&gt;&amp;1 |tee /tmp/SN134_Sync_log
* * * * * /usr/bin/perl /home/rpgamma/SN167-Rados/scritps/RadosSync_SN167pl 2&gt;&amp;1 |tee /tmp/SN167_Sync_log
</code></pre><hr><h4 id="Dependancies">Dependancies:</h4><p>The <a href="http://RadoSync.pl" target="_blank" rel="noopener">RadoSync.pl</a> Perl scripts running the Linux VM <strong>rados-server</strong> rely on the following PERL modules:</p><ul>
<li>Net::OpenSSH</li>
<li>HTTP::DAV</li>
<li>Crypt::SSLeay</li>
</ul><p>To install them, the following has been performed as root on the <strong>rados-server</strong>:</p><pre><code>yum install perl-cpan
cpan install Net::OpenSSH
cpan install HTTP::DAV
cpan install Crypt::SSLeay
</code></pre><hr><h4 id="User-Accounts-And-SSH-PrivatePublic-Key-Pairs">User Accounts And SSH Private/Public Key Pairs:</h4><ul>
<li>As the <strong><a href="http://RadosSync.pl" target="_blank" rel="noopener">RadosSync.pl</a></strong> scripts, relies on SSH and SCP for remote access to all of the <strong>rados units servers</strong> , the user account <strong>admin-rados</strong> had to have been created on each <strong>rados units servers</strong> , by performing as root on each: <strong>rados units servers</strong></li>
</ul><pre><code>passwd admin-rados
</code></pre><p>(Unlike on Linux, as the <strong>rados units servers</strong>  run on <a href="https://en.wikipedia.org/wiki/QNX" target="_blank" rel="noopener">QNX</a> (a UNIX version based NetBSD), the “passwd” command, both creates the user account and sets the password).</p><p>To perform the “password-less” copy using SCP of the report files, an SSH RSA key pair is used.</p><ul>
<li>
<p>On the <strong>rados-server</strong> the public / private keys are stored in <strong>rados-server:/home/rpgamma/.ssh/</strong>.</p>
</li>
<li>
<p>And on each <strong>rados units servers</strong>  the public key is stored in <strong>/home/admin-rados/.ssh/authorized_keys</strong>.</p>
</li>
</ul><p>To copy the public key on each <strong>rados mobile unit server</strong>, the following has been performed (as the <strong>rpgamma</strong> user account) for e`ach <strong>rados units servers</strong>:</p><pre><code>cat ~/.ssh/rados-mobil_rsa.pub | ssh admin-rados@&lt;rados_mobile_unit_server&gt; "mkdir -p ~/.ssh &amp;&amp; cat &gt;&gt; ~/.ssh/authorized_keys &amp;&amp; chmod 700 ~/.ssh &amp;&amp; chmod 600 ~/.ssh/authorized_keys"
</code></pre><p>To avoid having to specify where the private key is stored as parameter of the the SSH and SCP commands, the following SSH <strong>rados-server:/home/rpgamma/.ssh/config</strong> file has been created:</p><pre><code>Host rados*
    ServerAliveInterval 15

    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes
    StrictHostKeyChecking no

    user admin-rados
    IdentityFile ~/.ssh/rados-mobil_rsa
</code></pre><hr><h4 id="Variables">Variables:</h4><ul>
<li>
<p><strong>my $rado_mobile_server_name</strong> &#9664;&#65039; &#8210; – – The rados units server name from which we gonna copy the data. For example <strong>rados644</strong>.</p>
</li>
<li>
<p><strong>my $rado_mobile_server_user</strong> &#9664;&#65039; &#8210; – – The user name used to log on the rados units server (usually “admin-rados”).</p>
</li>
<li>
<p><strong>my $rado_mobile_server_transfma_dir</strong> &#9664;&#65039; &#8210; – – The base directory name where the source files are located (usually “/transfma”).</p>
</li>
<li>
<p><strong>my $target_dir</strong> &#9664;&#65039; &#8210; – – The target path where we copy the data. For example <strong>rados-server:/home/rpgamma/SN40-Rados/transfma</strong>.</p>
</li>
<li>
<p><strong>my $EOS_user</strong> &#9664;&#65039; &#8210; – – The user name used to log onto EOS (usually “rpgamma”).</p>
</li>
<li>
<p><strong>my $EOS_user_password</strong> &#9664;&#65039; &#8210; – – The user’s password used to log onto EOS.</p>
</li>
<li>
<p><strong>my $EOS_transfma_dir</strong> &#9664;&#65039; &#8210; – – The base EOS directory name where the reports are going to be backed up into (usually “/transfma”).</p>
</li>
<li>
<p><strong>my $sleepTime</strong> &#9664;&#65039; &#8210; – – The time the script waits before polling again it’s <strong>rados units server</strong> (default 60 seconds).</p>
</li>
</ul><hr><h4 id="Debugging-Options">Debugging Options:</h4><p>As hinted from the cron job each script output their respective log files such as: <strong>/tmp/SN40_Sync_log</strong>, <strong>/tmp/SN134_Sync_log</strong> and <strong>/tmp/SN167_Sync_log</strong>, each outputting:</p><ul>
<li>
<p>The rados units servers such as: rados644 matched with SN40-Rados, radosmobil matched with SN134-Rados and Rados661 matched with SN167-Rados.</p>
</li>
<li>
<p>The Target dirs such as: SN40-Rados, SN134-Rados and SN167-Rados</p>
</li>
<li>
<p>The Total amount of Files which important in comparison with the Total Files amount of files processed, Total of New Files, the amount of Copied and skipped Files (those that have already been copied). Should the figures not adding up, points to an issue that has to be investigated (see how to increasing the debugging level below).</p>
</li>
<li>
<p>The full source Path/FileName of the new file to be copied from the <strong>rados unit server</strong>.</p>
</li>
<li>
<p>Any newly discovered source subdirectory that doesn’t already exist on the target directories and is created on the <strong>rados-server</strong> and on the CERN <strong>/eos/project/h/hse-rp-cs/RadioLab/</strong> Linux/Windows workspace.</p>
</li>
<li>
<p>The name of the new file prefixed with the date stamp.</p>
</li>
<li>
<p>The full target Path/FileName (prefixed with the date stamp) to be copied on the <strong>rados-server</strong> and on the CERN <strong>/eos/project/h/hse-rp-cs/RadioLab/</strong> Linux/Windows workspace.</p>
</li>
<li>
<p>Whether or not the file doesn’t exist on on the <strong>rados-server</strong> and on the CERN <strong>/eos/project/h/hse-rp-cs/RadioLab/</strong> Linux/Windows workspace and in such a case having copied it.</p>
</li>
<li>
<p>The total of Total Files Processed, new files, copied and skipped files.</p>
</li>
</ul><p>To view the logs you can either type: <code>tail -f /tmp/&lt;logname&gt;</code> or <code>less /tmp/&lt;logname&gt;</code>.</p><h5 id="Increasing-the-debugging-level">Increasing the debugging level</h5><p>To find out at which step a problem is occurring, you can un-comment (by removing the # in front of a line of code) any line ending with <strong># for debuging</strong> and any line like <strong>## print “sleep\n”; sleep 5; # for debuging</strong>, then save the script and run it manually (ex: /home/rpgamma/&lt;target&gt;/script/script_name&gt; ).</p><hr><h4 id="The-Linux-VM-rados-server">The Linux VM rados-server</h4><p>As aforementioned, the <strong>rados-server</strong> is the server on which the <strong><a href="http://RadosSync.pl" target="_blank" rel="noopener">RadosSync.pl</a></strong> scripts runs and where the report files are copied first and as can be seen from the following Directory Structure, there is the same consistent directory structure for each <strong>rados units servers</strong> .</p><ul>
<li>
<p>A <strong>script</strong> directory containing a <strong><a href="http://RadosSync.pl" target="_blank" rel="noopener">RadosSync.pl</a></strong> script, matched to it’s corresponding <strong>rados units servers</strong>.</p>
</li>
<li>
<p>A <strong>transfma</strong> directory where the content of each <strong>rados units servers:/transfma/</strong> directory are copied into.</p>
</li>
</ul><h4 id="Directory-Structure">Directory Structure</h4><pre><code>rados-server:/
&#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; home/
              &#9492;&#9472;&#9472;&#9472; rpgamma/
                   &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; .ssh/
                   &#9474;       &#9500;&#9472;&#9472;&#9472; config
                   &#9474;       &#9500;&#9472;&#9472;&#9472; rados-mobil_rsa
                   &#9474;       &#9492;&#9472;&#9472;&#9472; rados-mobil_rsa.pub
                   &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; SN40-Rados/   &#9664;&#65039; &#8210; – – matched to server: rados644
                   &#9474;       &#9500;&#9472;&#9472;&#9472; script/
                   &#9474;       &#9474;    &#9492;&#9472;&#9472;&#9472;&#9472; RadosSync_SN40.pl
                   &#9474;       &#9492;&#9472;&#9472;&#9472; transfma/
                   &#9474;            &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; data file1
                   &#9474;            &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; data file2
                   &#9474;            &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; and so on ...
                   &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; SN134-Rados/   &#9664;&#65039; &#8210; – – matched to server: radosmobil
                   &#9474;       &#9500;&#9472;&#9472;&#9472;&#9472; script/
                   &#9474;       &#9474;     &#9492;&#9472;&#9472;&#9472;&#9472; RadosSync_SN134.pl
                   &#9474;       &#9492;&#9472;&#9472;&#9472;&#9472; transfma/
                   &#9474;             &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; data file1
                   &#9474;             &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; data file2
                   &#9474;             &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; and so on ...
                   &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; SN167-Rados/   &#9664;&#65039; &#8210; – – matched to server: Rados661
                           &#9500;&#9472;&#9472;&#9472;&#9472; script/
                           &#9474;     &#9492;&#9472;&#9472;&#9472;&#9472; RadosSync_SN167pl
                           &#9492;&#9472;&#9472;&#9472;&#9472; transfma/
                                 &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; dir1
                                 &#9474;       &#9492;&#9472;&#9472;&#9472; data file1
                                 &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; data file1
                                 &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; and so on ...
</code></pre><hr><h4 id="The-RADOS-units-servers">The RADOS units servers:</h4><ul>
<li>The <strong>rados units servers</strong> are the servers on which the reports of analyzed radio-active material are created.</li>
</ul><h4 id="RADOS-Units-Servers-Directory-Structure">RADOS Units Servers Directory Structure</h4><pre><code>rados mobile unit server:/
&#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; home/
                         &#9500;&#9472;&#9472;&#9472; admin-rados/
                         &#9474;    &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; .ssh/
                         &#9474;            &#9492;&#9472;&#9472;&#9472; authorized_keys
                         transfma/
                         &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; dir1
                         &#9474;       &#9492;&#9472;&#9472;&#9472; data file1
                         &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; data file1
                         &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; and so on ...
</code></pre><hr><h4 id="RADOS-EOS-Directory-Structure">RADOS EOS Directory Structure</h4><p>The EOS Directory Structure into which the reports are finally backed up is consistently organized like the <strong>rados-server:/home/rpgamma/</strong> Directory Structure.</p><pre><code>/eos/project/h/hse-rp-cs/RadioLab/
&#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; rpgamma/
                                 &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; SN40-Rados/
                                 &#9474;       &#9500;&#9472;&#9472;&#9472; script/
                                 &#9474;       &#9474;    &#9492;&#9472;&#9472;&#9472;&#9472; SN40_Sync.pl
                                 &#9474;       &#9492;&#9472;&#9472;&#9472; transfma/
                                 &#9474;            &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; data file1
                                 &#9474;            &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; data file2
                                 &#9474;            &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; and so on ...
                                 &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; SN134-Rados/
                                 &#9474;       &#9500;&#9472;&#9472;&#9472;&#9472; script/
                                 &#9474;       &#9474;     &#9492;&#9472;&#9472;&#9472;&#9472; SN134_Sync.pl
                                 &#9474;       &#9492;&#9472;&#9472;&#9472;&#9472; transfma/
                                 &#9474;             &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; data file1
                                 &#9474;             &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; data file2
                                 &#9474;             &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; and so on ...
                                 &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; SN167-Rados/
                                         &#9500;&#9472;&#9472;&#9472;&#9472; script/
                                         &#9474;     &#9492;&#9472;&#9472;&#9472;&#9472; SN167_Sync.pl
                                         &#9492;&#9472;&#9472;&#9472;&#9472; transfma/
                                               &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; dir1
                                               &#9474;       &#9492;&#9472;&#9472;&#9472; data file1
                                               &#9500;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; data file1
                                               &#9492;&#9472;&#9472;&#9472;&#9472;&#9472;&#9472; and so on ...
</code></pre></div>
Topic revision: r1 - 2020-01-21 - DominiqueCressatti
Account
- Log In
- Cern Search
- TWiki Search
- Google Search
Sandbox All webs
Edit
Attach
Copyright &© 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback