Data Transfers

Grid users with a valid certificate can first load their certificate into their browser, then register in the DREAM VOMS service to become registered members of the DREAM VO. Once registered, they can use grid data transfer tools to move data into and out of the storage area at TTU or other grid locations, such as CERN, where the DREAM VO is supported.

It is also possible to move data to and from TTU or other storage using the Globus Online service. This service has a downloadable stand-alone utility called "Globus Connect" that can be installed at the user's discretion on any laptop, desktop or other cluster system. There is a pre-registered instance of Globus Connect on the PCDREAM system at the test beam. Once installed and running, the functions of the data transfer can be accessed via command-line scripts or through the Globus Online service web pages. (See information in the control room for the appropriate GO access credentials.)

Instructions on how to use these tools for regular users are in the GlobusOnline portion of this TWiki. Required setup before doing anything using grid tools is described in the GridSetup pages. The portions below are for the support experts.

-- AlanSill - 18-Jun-2011

Anyone in DREAM: How to transfer files

(1) Log in to with dreamdaq account (ask Alan, Sehwook or Michele for the password).
(2) As soon as you log in, a new page will showed up and there will be "File Transfer", "My Profile" and "Globus Connect" links in the middle of the page. If this is the first time you use Globus Online then click on "Globus Connect" to download the GO client to your pc and follow the instructions to register your machine.
(3) Click "File Transfer" and very similar page like a window of a ftp program will show up.
(3) Select dreamdaq#TTU-Sigmorgh-DREAM as one of the endpoints and your pc as the other (if the dreamdaq#TTU-Sigmorgh-DREAM endpoint is not available contact Alan, Sehwook or Michele).
(4) Go to the directory where the files you need are located and select them (you can select more then one by holding Ctrl and clicking on them)
(5) On the other window select the folder where you want the filse to be copied to.
(6) Click the arrow button between two file lists to start the transfer.
(7) A message will appear on top of the page.

How to get user certificate and key files. Follow the instruction for your region of the globe as described on the following site.

Experts only: Activate the TTU-Sigmorgh-DREAM endpoint

(These should be done from your own personal account, not root or the dreamtest one).

(1) Log in to with dreamdaq account.
(2) Click "My Profile" and then go to "Manage Identities" on the left side of the web page.
(3) Click "Add X.509 Certificate".
(4) See GridHelp if you need a certificate. Copy and paste whole content of your ~/.globus/usercert.pem file to the "X.509 Certificate (PEM Encoded)" box which is prompted when you click "Add X.509 Certificate". For the "Alias" box, please refer to the other persons' aliases.
The steps from (1) to (4) are needed only once when you register your X.509 certificate to the globusonline.
(5) Now, You should have the user certificate and key files in the .globus directory at your home directory for the TTU HPCC machines.
(6) At any machine equipped with grid software (e.g., etc.), on bash

voms-proxy-init -voms dream:/dream/dreamdaq -hours 168 (you need to type your GRID pass phrase that you set when you created your user certificate and key files)
gsissh -t endpoint-activate -g TTU-Sigmorgh-DREAM

(Remember to remove the SPAMNOT from the above line if you copy and paste it; that is put in by the TWIki software and is not part of the command.)

Once the above has been done, the following can be done by anyone to copy data from the control room to TTU storage, and similar commands can be used to copy data from the activated endpoint to another remote location as described in the GlobusOnline page. See the DreamDataAnalysis page for more on data analysis operations.

    • ~Daily TTU mirror procedure: Note: REQUIRES the TTU endpoint to be activated by Michele, Sehwook or Alan by grid proxy ahead of time to work
    • Login to the dreamtest account on pcdreamdaq2, or to any other account that has ssh keys stored in the Globus dreamdaq identity store
    • Execute: ssh -t scp -r -s 0 -v pcdreamdaq2personal:storage/ TTU-Sigmorgh-DREAM:/lustre/hep/osg/dream/2015/

Experts Only: Log in to TTU HPCC machine to manage files remotely via dreamdaq role

(Do this from your personal account, not dreamtest)

(1) You also need the user certificate an key files in the .globus directory at your home directory.
(2) Execute the following commands on bash

(NOTE: the extra stuff before the voms-proxy-init command is not needed on lxplus, where the DREAM VO is already defined.)

voms-proxy-init -voms dream:/dream/dreamdaq (you need to type your GRID pass phrase that you set when you created your user certificate and key files)
gsissh -p 49922
gsissh -p 49922

Now, you are logged in to a TTU HPCC machine with your grid proxy mapped to the dreamdaq account.

-- SehwookLee - 22-Aug-2011
-- MicheleCascella - 16-Sep-2011
-- AlanSill - 2015-10-20

Edit | Attach | Watch | Print version | History: r15 < r14 < r13 < r12 < r11 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r15 - 2017-07-18 - AlanSill
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    DREAM All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback