Transfer Watch Procedure

In general, we use the data transfer management system, PhEDEx, to do transfer watch. We usually don't care about T3 transfers. Below is the primary steps we are following in the watch:

PhEDEx Transfers

<1a> Quality Plots :

• Check transfer quality from T0|T1|T2 to T1|T2 in the last 24 hours : https://cmsweb.cern.ch/phedex/prod/Activity::QualityPlots?graph=quality_all&entity=link&src_filter=T0|T1|T2&dest_filter=T0|T1|T2&period=l24h
• Red links:
  • They are failing transfers, and we should further check their transfer errors.
  • For example, if we find a red link from FNAL to Bari. The next thing is to check all transfers to Bari and all transfers from FNAL.
  • T0|T1|T2 -> Bari: https://cmsweb-testbed.cern.ch/phedex/prod/Activity::QualityPlots?graph=quality_all&entity=link&src_filter=T0|T1|T2&dest_filter=Bari&no_mss=true&period=l24h
  • FNAL -> T0| T1|T2: https://cmsweb-testbed.cern.ch/phedex/prod/Activity::QualityPlots?graph=quality_all&entity=link&src_filter=FNAL&dest_filter=T0|T1|T2&no_mss=true&period=l24h
  • By those double check all transfers from the SOURCE and all transfers to the DESTINATION, we usually could guess the problematic site.(not always)
  • Then we need to further check the detailed transfer errors that caused transfer fails. <2>
• White links:
  • They indicates that there should be some transfers going on, but for some reason they are stuck somewhere, neither failing, nor proceeding
  • (If the line is not completely white, that can also mean there is no transfer at that time period)
  • Then we need to go the transfer Rate page to further check. <3>

<1b> T1 Disk Latency Overview :

• https://transferteam.web.cern.ch/transferteam/LatencyOverview/
• This page shows all subscriptions to T1_*_Disk endpoints
• Check current subscriptions, if "Estimated Arrival Time" is NA, that means this transfer has problems
• To see what the problem is, you can click problematic subscription, and see the reason of each dataset's problem by Problem column
• Please note that preprocessing subscriptions are more important than other subscriptions to Disk endpoints. Please check "notes" section in this page.

<2> Recent Errors

• https://cmsweb.cern.ch/phedex/prod/Activity::ErrorInfo
• We can filter the transfer errors of the problematic link by inputting the site name in this page
• Then check the detail log, it usually gives the name of the error, and tells you at which site there are errors.(not always)
• Thus we open a GGUS ticket to the problematic site, and ask them to check. <6>

<3> Rate

• https://cmsweb.cern.ch/phedex/prod/Activity::Rate
• Check if there are expiring transfers related to the site.
• Further check the transfer detail if transfer stuck in some state. <4>

<4> Transfer Details

• https://cmsweb.cern.ch/phedex/prod/Activity::TransferDetails
• Check transfer detail at this moment.
• If there are files stuck in assigned state. But we don't see a problem there. It is a candidate to open a ticket.
• If many files are transferring at this moment, we would wait a bit and see how it goes, maybe site people noticed problem themselves, fixed already, and now everything should be fine.

<5> Routing

• https://cmsweb.cern.ch/phedex/prod/Activity::Routing
• It shows if there are problems with some particular blocks.

ASO Transfers

<1c> ASO Transfer Monitoring :

• Monitoring page currently only works with Firefox
• Check Async transfer activity and quality in the last 24 hours here: https://cmsweb.cern.ch/aso-monitor/index.html
• In case there are failed transfers check the source-destination matrix: https://cmsweb.cern.ch/aso-monitor/njobDS.html
  • By clicking on red dots (the size of the dots is proportional to the number of failures) one can see Destination (Source) monitoring in the last 24 hours and pinpoint if the failure is at Destination (Source) site.
  • Currently no easy way to obtain FTS logs for a particular site. Nevertheless one can get the FTS logs for failed transfers by going to https://cmsweb.cern.ch/aso-monitor/totbyWorkflow.html to get the failed workflow name and finally going to https://cmsweb.cern.ch/aso-monitor/ftsLogs.html to get the logs by the workflow name.
  • Check in COUCH the site, dest, date, failure reason, user, workflow, and lfn. https://cmsweb.cern.ch/couchdb/_utils/database.html?asynctransfer/_design/AsyncTransfer/_view/JobsByFailuresReasons
• Open a GGUS ticket to the problematic site, and ask them to check. <6>

---

<6> GGUS Ticket

• https://ggus.eu/?mode=ticket_cms
• Before open a ticket check if the site is in downtime: http://cmsdoc.cern.ch/cms/FacOps/sls/gcal/index.html
• Before open a ticket check if there is already a ticket opened for this issue:
  • GGUS ticket search

<7> Transfer Trouble Shooting

• This page helps us to debug the transfer issues, although we are not responsible for debugging transfer problems, it helps us to do transfer watch.
  • https://twiki.cern.ch/twiki/bin/view/CMS/PhedexSiteAdminTransferTroubleshooting
• Some twiki pages for common transfer errors:
  • https://twiki.cern.ch/twiki/bin/view/LCG/TransferOperationsPopularErrors
  • https://twiki.cern.ch/twiki/bin/view/CMSPublic/CompOpsTransferTeamBasicConceptsTransferErrors
  • https://twiki.cern.ch/twiki/bin/view/CMSPublic/DataTransferTroubleshootingGuide