Current issues on CERN-PROD FTS 2.0 service

> ftscp(6196): Reason: The process serving the transfer (status = 
> RUNNING) is no longer active (could not open file
> /proc/5734/cmdline) 2007-06-20 13:49:44 ftscp(6196):

The error is the same error as you get on FTS 1.5 servers:

"No job found for this ID"

(we have to pending improvement to make the error message useful..)


What it means is that the dCache gridFTP server at PIC sent a non-gridFTP protocol compliant message 
to the globus client running the transfer (actually, I think there is a race condition in the girdFTP 1 
protocol itself). There is a bug (or mis-feature) in the the current VDT/Globus library we use, 
in that if it receives such a bad message, it calls abort() in the library routine (rather than just failing) 
- and this causes our transfer process to die uncleanly without being able to log much useful 
information. We should make this more obvious in the error message. This is currently 
associated with one particular gridFTP server door at PIC. (dc003 I think).

Presumably similar to "Bad error message for dead transfers" issue.

Another bad (but not quite the same) message from one of the gridFTP 
servers triggers another bug in the Globus gridFTP client library we use, causing 
it to go into an infinite loop deep inside the library copy call. This would 
normally not be bad (since we should just kill the bad process and retry), 
but a bug in our agent causes it to block while trying to kill such processes.


Our immediate course of action is to fix the problem in our agent 
- so these bad transfers gets killed cleanly and we don’t block the channel.

Then we can look at the problems with the Globus client library we use 
- some of the bugs we know are fixed in the latest VDT 1.6 version of Globus, 
so we plan to migrate to that - as soon as we've run some stress tests with it 
on the FTS pilot service.

Also, I tried to cancel a long-lasting transfer. It is still pending:

glite-transfer-status -s
https://prod-fts-ws.cern.ch:8443/glite-data-transfer-fts/services/FileTransfer
94df5e40-1f30-11dc-840f-a3448bce587f -l

about 10 minutes ago (2007-06-20 17:09:46 CERN Time), it is still trying to cancel it (Canceling).

Could you please have a look on this?


Thanks again,
Yujun