I. glite-wms-job-submit -a --nolisten j.jdl JDL file: JobType = "Interactive"; Executable ="j.sh"; InputSandbox = {"j.sh"}; ShallowRetryCount=5; Requirements=RegExp(".*lxb20.*",other.GlueCEUniqueID); Executable: #!/bin/bash echo "Wellcome!" read name echo "Bye, $name" exit 0 - The glite-wms-grid-console-shadow process is appeared right now after launching command. - Pipes are created. - When job has been successfully submitted to the WMProxy(command output) : >====================== glite-wms-job-submit Success ====================== > >The job has been successfully submitted to the WMProxy >Your job identifier is: > >https://lxb2032.cern.ch:9000/wUW6bRR0SJvdA78yTaeZ4w > >Interactive Shadow Console successfully launched >With the following parameters: >- Host: pcitgd02.cern.ch >- Port: 21065 >- Shadow process Id: 2685 >- Input Stream location: /tmp/listener-NQghjxWaDn6ThIoG4wVUFA.in >- Output Stream location: /tmp/listener-NQghjxWaDn6ThIoG4wVUFA.out >*** Warning *** > Make sure you will kill the Shadow process > and remove the input/output streams when interaction finishes - After that command finished without any messages and there is no grid-console-shadow process in memory and there are no specified pipes. - Submitted job hangs up in the "Running" state. - The same command behavior on lxb1765,lxb2057 and pcitgd02 UI. - logfile for this session see in A1. ================================================================================== II. glite-wms-job-submit -a j.jdl ---------------------1. Executable: #!/bin/bash echo "Wellcome!" exit 0 - Shadow process and pipes are OK. - Get "Wellcome!" message from submitted job - Job terminated successfully - We have to press ^C to exit from session - !!!!!!!!!!! the pipe.in is not removed !!!!!!!!!!!! -------------------- 2. Executable: #!/bin/bash echo "Wellcome!" echo "Tell me your name" read name echo "Bye, $name" exit 0 - Shadow process and pipes are OK. - !!!!!!! No any output from the job until something will be hitting on keyboard: > Nik > Wellcome! > Tell me your name > Bye, Nik > > *********************************** > * INTERACTIVE JOB FINISHED * > *********************************** - Job terminated successfully - !!!!!!!!!!! the pipe.in is not removed -------------------- 3. Executable: #!/bin/bash echo "Wellcome!" echo "Tell me your name" read name echo "$name" > out.txt echo "Read $name" echo "Tell me surname" read surname echo "$surname" >> out.txt echo "Bye, name=$name, surname=$surname" printenv > env.txt exit 0 - Shadow process and pipes are OK. - !!!!!!! No any output from the job until something will be hitting on keyboard: > Nik > Wellcome! > Tell me your name > Read Nik > Tell me surname > Bye, name=Nik, surname= > > *********************************** > * INTERACTIVE JOB FINISHED * > *********************************** - Job terminated successfully - !!!!!!!! Second read in the executable read nothing. - Output from printenv on WN see in A2 !!!!!!!! Looking as all output from job is transfered only after job exit -------------------- 4. ^C when job is not in the state "Runnig" yet - !!!!!!!!!!!!!! command segmentation fault ================================================================================== III. glite-job-submit from UI lxb1765, lxb2057 and pcitgd02 - Shadow process and pipes are OK. !!!!!!!!!! No any output from the job, job hangs up in th state "Running" for lxb1765 and lxb2057 ------------ BYPASS_SHADOW_HOST (must be UI host): 1. lxb2057: BYPASS_SHADOW_HOST= 137.138.4.22 host 137.138.4.22 => 22.4.138.137.in-addr.arpa domain name pointer lxplus207.cern.ch ping from WN to 137.138.4.22 is OK !!!!!!!!!! No any output from the job, job hangs up in th state "Running" 2. lxb1765: BYPASS_SHADOW_HOST= 128.142.66.35 host 128.142.66.35 => Host 35.66.142.128.in-addr.arpa not found: 3(NXDOMAIN) ping from WN to 28.142.66.35 is failed. !!!!!!!!!! No any output from the job, job hangs up in th state "Running" 3. pcitgd02: BYPASS_SHADOW_HOST= 137.138.32.183 host 137.138.32.183 => 183.32.138.137.in-addr.arpa domain name pointer pcitgd02.cern.ch ping is OK Interactive session OK Note: command glite-wms-job-submit always set BYPASS_SHADOW_HOST as proper UI address (lxb1765.cern.ch, lxb1765.cern.ch or pcitgd02.cern.ch accordingly) !!!!! Conclusion: it seems ip address in /etc/hosts are wrong on lxb2057 and lxb1765: ssh lxb2057 @lxb2057]~%cat /etc/hosts @lxb2057]~% 137.138.4.22 lxb2057.cern.ch @lxb2057]~%traceroute lxb2057.cern.ch @lxb2057]~%traceroute to lxb2057.cern.ch (137.138.4.22), 30 hops max, 38 byte packets @lxb2057]~% 1 l513-v-rftec-1-ip167 (128.142.200.1) 1.530 ms 0.304 ms 0.298 ms @lxb2057]~% 2 l513-b-rftec-2-rl7 (194.12.138.9) 5.777 ms 0.397 ms 0.454 ms @lxb2057]~% 3 b513-b-rfte6-2-sl1 (194.12.131.73) 0.644 ms 12.734 ms 0.526 ms @lxb2057]~% 4 b513-c-rfte6-1-rb16 (194.12.132.62) 0.685 ms 12.784 ms 0.622 ms @lxb2057]~% 5 lxb2057 (137.138.4.22) 0.595 ms 0.646 ms 0.628 ms @lxb2057]~%traceroute 128.142.200.50 @lxb2057]~%traceroute to 128.142.200.50 (128.142.200.50), 30 hops max, 38 byte packets @lxb2057]~% 1 lxb2057 (128.142.200.50) 0.139 ms 0.084 ms 0.034 ms ssh lxb1765 cat /etc/hosts 128.142.66.35 lxb1765.cern.ch traceroute lxb1765.cern.ch traceroute to lxb1765.cern.ch (128.142.66.35), 30 hops max, 38 byte packets 1 l513-v-rftec-1-ip165 (128.142.199.1) 3.402 ms 0.310 ms 0.261 ms 2 * * * 3 * * * traceroute 128.142.199.48 traceroute to 128.142.199.48 (128.142.199.48), 30 hops max, 38 byte packets 1 lxb1765 (128.142.199.48) 0.135 ms 0.079 ms 0.035 ms ============================================================================== A1. logfile from the glite-wms-job-submit -a --nolisten --logfile log j.jdl: ******************************************************************************** WMS User Interface - version 1.2.1 - Log File ******************************************************************************** ------------------------------------------ 14 Feb 2007, 09:42:50 -I- PID: 2683 (Info) - Function: glite-wms-job-submit ------------------------------------------ 14 Feb 2007, 09:42:50 -I- PID: 2683 (Info) - Options: --autm-delegation; --nolisten; --logfile logwmsnolist; ------------------------------------------ 14 Feb 2007, 09:42:50 -I- PID: 2683 (Debug) - Vo read from proxy certificate extension ------------------------------------------ 14 Feb 2007, 09:42:50 -I- PID: 2683 (Debug) - VirtualOrganisation value : dteam ----------------------------------------- 14 Feb 2007, 09:42:50 -I- PID: 2683 (Debug) - Auto-Generation of the Delegation Identifier: zZWA47Wf9pWedxxuQqW_VA ----------------------------------------- 14 Feb 2007, 09:42:50 -I- PID: 2683 (Debug) - Getting Endpoint URL from configuration file ----------------------------------------- 14 Feb 2007, 09:42:50 -I- PID: 2683 (Info) - Connecting to the service https://lxb2032.cern.ch:7443/glite_wms_wmproxy_server ----------------------------------------- 14 Feb 2007, 09:42:50 -I- PID: 2683 (Debug) - Calling the WMProxy getVersion service ----------------------------------------- 14 Feb 2007, 09:42:53 -I- PID: 2683 (Debug) - WMProxy: major version[2] - minor version[1] ----------------------------------------- 14 Feb 2007, 09:42:53 -I- PID: 2683 (Debug) - getVersion - Version numbers successfully retrieved ----------------------------------------- 14 Feb 2007, 09:42:53 -I- PID: 2683 (Debug) - WMProxy Version: 2.1.0 ----------------------------------------- 14 Feb 2007, 09:42:53 -I- PID: 2683 (Debug) - Sending Proxy Request to https://lxb2032.cern.ch:7443/glite_wms_wmproxy_server ----------------------------------------- 14 Feb 2007, 09:42:53 -I- PID: 2683 (Debug) - Calling the WMProxy ns2__getProxyReq service ----------------------------------------- 14 Feb 2007, 09:42:54 -I- PID: 2683 (Debug) - ns2__getProxyReq - The proxy has been successfully retrieved ----------------------------------------- 14 Feb 2007, 09:42:54 -I- PID: 2683 (Debug) - Calling the WMProxy ns2__putProxyReq service ----------------------------------------- 14 Feb 2007, 09:42:55 -I- PID: 2683 (Debug) - ns2__putProxyReq - The proxy has been successfully delegated with the identif ier: zZWA47Wf9pWedxxuQqW_VA ----------------------------------------- 14 Feb 2007, 09:42:55 -I- PID: 2683 (Debug) - The proxy has been successfully delegated with the identifier: zZWA47Wf9pWedxx uQqW_VA ----------------------------------------- 14 Feb 2007, 09:42:55 -I- PID: 2683 (Debug) - The JDL file is: /home/klopov/tests/j.jdl ----------------------------------------- 14 Feb 2007, 09:42:55 -I- PID: 2683 (Debug) - File archiving and file compression allowed by user in the JDL ----------------------------------------- 14 Feb 2007, 09:42:55 -I- PID: 2683 (Debug) - An interactive job is being submitted. ----------------------------------------- 14 Feb 2007, 09:42:55 -I- PID: 2683 (Debug) - Running console shadow ----------------------------------------- 14 Feb 2007, 09:43:00 -I- PID: 2683 (Debug) - Console properly started ----------------------------------------- 14 Feb 2007, 09:43:00 -I- PID: 2683 (Debug) - Calling the WMProxy getFreeQuota service ----------------------------------------- 14 Feb 2007, 09:43:00 -I- PID: 2683 (Debug) - getFreeQuota - User freequota not set on the server ----------------------------------------- 14 Feb 2007, 09:43:00 -I- PID: 2683 (Debug) - Getting the max ISB size from the server https://lxb2032.cern.ch:7443/glite_wm s_wmproxy_server ----------------------------------------- 14 Feb 2007, 09:43:00 -I- PID: 2683 (Debug) - Calling the WMProxy getMaxInputSandboxSize service ----------------------------------------- 14 Feb 2007, 09:43:01 -I- PID: 2683 (Debug) - getMaxInputSandboxSize - Max ISB size information successfully retrieved ----------------------------------------- 14 Feb 2007, 09:43:01 -I- PID: 2683 (Debug) - The InputSandbox size (65 bytes) doesn't exceed the max size limit of 10000000 bytes: File transfer is allowed ----------------------------------------- 14 Feb 2007, 09:43:01 -I- PID: 2683 (Debug) - Registering JDL [ requirements = ( ( RegExp(".*lxb20.*",other.GlueCEUniqueID) ) && ( other.GlueCEStateStatus == "Production" ) && other.GlueHostNetworkAdapterOutboundIP ) && ( other.GlueCEStateStatus == "Production" ) && other.GlueHostNetworkAdapterOutboundIP; ListenerHost = "pcitgd02.cern.ch"; ListenerPipeName = "/tmp/liste ner-NQghjxWaDn6ThIoG4wVUFA"; RetryCount = 3; MyProxyServer = "lxb2020.cern.ch"; AllowZippedISB = true; JobType = "interactiv e"; Executable = "j.sh"; ListenerPort = 21065; VirtualOrganisation = "dteam"; rank = -other.GlueCEStateEstimatedResponseTim e; Type = "job"; ShallowRetryCount = 5; DefaultRank = -other.GlueCEStateEstimatedResponseTime; ZippedISB = { "ISBfiles__4fO 1EgGKjB-EJAMmCU6Ag_0.tar.gz" }; InputSandbox = { "file:///home/klopov/tests/j.sh" } ] ----------------------------------------- 14 Feb 2007, 09:43:01 -I- PID: 2683 (Debug) - Registering the job to the service https://lxb2032.cern.ch:7443/glite_wms_wmpr oxy_server ----------------------------------------- 14 Feb 2007, 09:43:01 -I- PID: 2683 (Debug) - Calling the WMProxy jobRegister service ----------------------------------------- 14 Feb 2007, 09:43:07 -I- PID: 2683 (Debug) - The job has been successfully registered ----------------------------------------- 14 Feb 2007, 09:43:07 -I- PID: 2683 (Debug) - The JobId is: https://lxb2032.cern.ch:9000/wUW6bRR0SJvdA78yTaeZ4w ----------------------------------------- 14 Feb 2007, 09:43:07 -I- PID: 2683 (Debug) - Getting the SandboxBulkDestinationURI from the service https://lxb2032.cern.ch :7443/glite_wms_wmproxy_server ----------------------------------------- 14 Feb 2007, 09:43:07 -I- PID: 2683 (Debug) - Calling the WMProxy getSandboxBulkDestURI service ----------------------------------------- 14 Feb 2007, 09:43:09 -I- PID: 2683 (Debug) - getSandboxBulkDestURI - Destination URIs sucessfully retrieved ----------------------------------------- 14 Feb 2007, 09:43:09 -I- PID: 2683 (Debug) - DestinationURI: gsiftp://lxb2032.cern.ch:2811/var/glite/SandboxDir/wU/https_3 a_2f_2flxb2032.cern.ch_3a9000_2fwUW6bRR0SJvdA78yTaeZ4w/input ----------------------------------------- 14 Feb 2007, 09:43:09 -I- PID: 2683 (Debug) - Archiving the ISB files: /tmp/ISBfiles__4fO1EgGKjB-EJAMmCU6Ag_0.tar ----------------------------------------- 14 Feb 2007, 09:43:09 -I- PID: 2683 (Debug) - This archive file has been successfully created: /tmp/ISBfiles__4fO1EgGKjB-EJA MmCU6Ag_0.tar ----------------------------------------- 14 Feb 2007, 09:43:09 -I- PID: 2683 (Debug) - Compressing the file (.gz): /tmp/ISBfiles__4fO1EgGKjB-EJAMmCU6Ag_0.tar ----------------------------------------- 14 Feb 2007, 09:43:09 -I- PID: 2683 (Debug) - ISB ZIPPED file successfully created: /tmp/ISBfiles__4fO1EgGKjB-EJAMmCU6Ag_0.t ar.gz ----------------------------------------- 14 Feb 2007, 09:43:09 -I- PID: 2683 (Debug) - FileTransfer (gsiftp): using globus-url-copy to transfer the local InputSandBo x file(s) to the submission endpoint ----------------------------------------- 14 Feb 2007, 09:43:09 -I- PID: 2683 (Debug) - File Transfer (gsiftp) /home/klopov/gLite/globus/bin/globus-url-copy file:///tmp/ISBfiles__4fO1EgGKjB-EJAMmCU6Ag_0.tar.gz gsiftp://lxb2032.cern.ch :2811/var/glite/SandboxDir/wU/https_3a_2f_2flxb2032.cern.ch_3a9000_2fwUW6bRR0SJvdA78yTaeZ4w/input/ISBfiles__4fO1EgGKjB-EJAMm CU6Ag_0.tar.gz ----------------------------------------- 14 Feb 2007, 09:43:10 -I- PID: 2683 (Debug) - File Transfer (gsiftp) TRANSFER DONE ----------------------------------------- 14 Feb 2007, 09:43:10 -I- PID: 2683 (Debug) - Starting the job: https://lxb2032.cern.ch:9000/wUW6bRR0SJvdA78yTaeZ4w ----------------------------------------- 14 Feb 2007, 09:43:10 -I- PID: 2683 (Debug) - Calling the WMProxy jobStart service ----------------------------------------- 14 Feb 2007, 09:43:11 -I- PID: 2683 (Debug) - jobStart - The job has been successfully started ==================================================================================================================== A2. printenv HOSTNAME=lxb0731.cern.ch GLITE_SD_PLUGIN=bdii,rgma BYPASS_SHADOW_HOST=pcitgd02.cern.ch VO_ALICE_SW_DIR=/opt/exp_soft/alice SHELL=/bin/sh LCG_LOCATION_VAR=/opt/lcg/var HISTSIZE=1000 CATALINA_HOME=/var/lib/tomcat5 PBS_JOBNAME=STDIN GLOBUS_LOCATION=/opt/globus GLOBUS_PATH=/opt/globus PERL5LIB=/opt/glite/lib/perl5:/opt/gpt/lib/perl VO_OPS_DEFAULT_SE=lxb1921.cern.ch GRID_CONSOLE_RETRY_TIMEOUT=6 DPM_HOST=lxb1921.cern.ch GLOBUS_GASS_CACHE_DEFAULT=/home/dteam015/globus-tmp.lxb0731.12844.0/globus-tmp.lxb0731.12844.0 PBS_ENVIRONMENT=PBS_BATCH EDG_TMP=/tmp GLITE_WMS_LOCATION=/opt/glite PBS_O_WORKDIR=/home/dteam015 GLITE_LOCATION_LOG=/var/log/glite GLOBUS_CE=lxb2018.cern.ch:2119/jobmanager-lcgpbs-dteam USER=dteam015 PBS_TASKNUM=1 LS_COLORS= LD_LIBRARY_PATH=.:/opt/lcg/lib:/opt/globus/lib:/opt/glite/lib:/opt/glite/externals/lib:/usr/local/lib:/opt/edg/lib:/usr/loca l/lib:/opt/glite/lib:/opt/glite/externals/lib:/opt/d-cache/dcap/lib:/opt/globus/lib MYPROXY_TCP_PORT_RANGE=20000,25000 JAVA_INSTALL_PATH=/usr/java/j2sdk1.4.2_08 PBS_O_HOME=/home/dteam015 GPT_LOCATION=/opt/gpt LCG_LOCATION=/opt/lcg VO_CMS_SW_DIR=/opt/exp_soft/cms VO_DTEAM_SW_DIR=/opt/exp_soft/dteam GLITE_LOCATION_TMP=/tmp/glite EDG_WL_TMP=/var/edgwl CLASSADJ_INSTALL_PATH=/usr PBS_MOMPORT=15003 LIBPATH=/opt/globus/lib:/usr/lib:/lib VO_LHCB_DEFAULT_SE=lxb1921.cern.ch GRID_CONSOLE_STDOUT=/tmp/listener-gKceUlyOMlQIDn89rxqjDA.out VO_OPS_SW_DIR=/opt/exp_soft/ops VO_ATLAS_DEFAULT_SE=lxb1921.cern.ch EDG_WL_USER=edguser GLITE_WMS_JOBID=https://lxb2032.cern.ch:9000/TZseJG7ifh--nT2iL3ORTQ GLOBUS_GRAM_MYJOB_CONTACT=URLx-nexus://lxb2018.cern.ch:20003/ PBS_O_QUEUE=dteam GLITE_WMS_RB_BROKERINFO=/home/dteam015/globus-tmp.lxb0731.12844.0/https_3a_2f_2flxb2032.cern.ch_3a9000_2fTZseJG7ifh--nT2iL3O RTQ/.BrokerInfo PATH=/opt/globus/bin:/opt/globus/sbin:/usr/java/j2sdk1.4.2_12/bin:/opt/lcg/bin:/usr/kerberos/bin:/opt/glite/bin:/opt/glite/e xternals/bin:/usr/java/j2sdk1.4.2_12/bin::/usr/bin:/opt/edg/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/opt/glite/bin:/ opt/glite/externals/bin:/opt/gpt/sbin:/opt/d-cache/srm/bin:/opt/d-cache/dcap/bin:/opt/edg/sbin:/home/dteam015/globus-tmp.lxb 0731.12844.0/https_3a_2f_2flxb2032.cern.ch_3a9000_2fTZseJG7ifh--nT2iL3ORTQ/ PBS_O_LOGNAME=dteam015 MAIL=/var/spool/mail/dteam015 PBS_O_LANG=C DPNS_HOST=lxb1921.cern.ch EDG_WL_LOCATION=/opt/edg PBS_JOBCOOKIE=03FF8D6560466DCF7F3FA194CC831291 EDG_LOCATION=/opt/edg LCG_TMP=/tmp VO_CMS_DEFAULT_SE=lxb1921.cern.ch VO_DTEAM_DEFAULT_SE=lxb1921.cern.ch PWD=/home/dteam015/globus-tmp.lxb0731.12844.0/https_3a_2f_2flxb2032.cern.ch_3a9000_2fTZseJG7ifh--nT2iL3ORTQ INPUTRC=/etc/inputrc JAVA_HOME=/usr/java/j2sdk1.4.2_12 SITE_GIIS_URL=lxb2018.cern.ch LANG=en_US.UTF-8 GLOBUS_REMOTE_IO_URL=/tmp/globus-tmp.lxb0731.12844.2 PBS_NODENUM=0 SASL_PATH=/opt/globus/lib/sasl PBS_O_SHELL=/bin/bash PERLLIB=/opt/lcg/lib/perl:/opt/edg/lib/perl:/opt/glite/lib/perl5 MYPROXY_SERVER=lxb2020.cern.ch GLITE_WMS_SEQUENCE_CODE=UI=000000:NS=0000000007:WM=000009:BH=0000000000:JSS=000006:LM=000006:LRMS=000003:APP=000000 UI=000000:NS=0000000007:WM=000009:BH=0000000000:JSS=000006:LM=000006:LRMS=000002:APP=000000 PBS_JOBID=2763.lxb2018.cern.ch APEL_HOME=/opt/glite ENVIRONMENT=BATCH VO_LHCB_SW_DIR=/opt/exp_soft/lhcb EDG_WL_LOCATION_VAR=/opt/edg/var HOME=/home/dteam015/globus-tmp.lxb0731.12844.0 SHLVL=8 GLITE_LOCATION_VAR=/var/glite VO_ALICE_DEFAULT_SE=lxb1921.cern.ch GLOBUS_TCP_PORT_RANGE=20000 25000 X509_USER_PROXY=/tmp/globus-tmp.lxb0731.12844.0 PBS_O_HOST=lxb2018.cern.ch COG_INSTALL_PATH=/usr SCRATCH_DIRECTORY=/home/dteam015/globus-tmp.lxb0731.12844.1 PBS_VNODENUM=0 EDG_LOCATION_VAR=/opt/edg/var GLITE_ENV_SET=true LOGNAME=dteam015 PYTHONPATH=/opt/lcg/lib:/opt/lcg/lib/python:/opt/glite/lib/python:/opt/edg/lib:/opt/edg/lib/python LCG_GFAL_INFOSYS=lxb2017.cern.ch:2170 PBS_QUEUE=dteam LD_FAKE_PRELOAD=./libglite-wms-grid-console-agent.so.0 LESSOPEN=|/usr/bin/lesspipe.sh %s RGMA_HOME=/opt/glite SHLIB_PATH=/opt/globus/lib GRID_CONSOLE_STDIN=/tmp/listener-gKceUlyOMlQIDn89rxqjDA.in VO_ATLAS_SW_DIR=/opt/exp_soft/atlas LOG4J_INSTALL_PATH=/usr GLITE_LOCATION=/opt/glite GLOBUS_GRAM_JOB_CONTACT=https://lxb2018.cern.ch:20002/21947/1171449322/ SITE_NAME=sl3certtb1a PBS_NODEFILE=/var/spool/pbs/aux//2763.lxb2018.cern.ch G_BROKEN_FILENAMES=1 SRM_PATH=/opt/d-cache/srm PBS_O_PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin GLITE_WMS_LOG_DESTINATION=lxb2018.cern.ch BYPASS_SHADOW_PORT=24246 ========================================================================================