cmsgwms-submit1

JulianBadillo - 2015-12-15

Jobs in Condor

[cmsdataops@cmsgwms-submit1 current]$ condor_q
-- Schedd: cmsgwms-submit1.fnal.gov : <131.225.205.6:9615?...
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               

0 jobs; 0 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended

Jobs Queued

MariaDB [wmagent]> SELECT wmbs_job.id AS id, wmbs_job.name AS name,
wmbs_job.cache_dir AS cache_dir,
wmbs_sub_types.name AS type, wmbs_job.retry_count AS retry_count,
wmbs_subscription.workflow as workflow,
wmbs_subscription.last_update as timestamp,
wmbs_workflow.name as request_name,
wmbs_workflow.priority as task_priority,
wmbs_workflow.task as task_name
FROM wmbs_job
INNER JOIN wmbs_jobgroup ON
wmbs_job.jobgroup = wmbs_jobgroup.id
INNER JOIN wmbs_subscription ON
wmbs_jobgroup.subscription = wmbs_subscription.id
INNER JOIN wmbs_sub_types ON
wmbs_subscription.subtype = wmbs_sub_types.id
INNER JOIN wmbs_job_state ON
wmbs_job.state = wmbs_job_state.id
INNER JOIN wmbs_workflow ON
wmbs_subscription.workflow = wmbs_workflow.id
WHERE wmbs_job_state.name = 'created';
Empty set (0.07 sec)

MariaDB [wmagent]> select wmbs_job_state.name, count(*)
from wmbs_job
join wmbs_job_state on (wmbs_job.state = wmbs_job_state.id)
group by wmbs_job.state, wmbs_job_state.name;
+----------+----------+
| name     | count(*) |
+----------+----------+
| cleanout |    32082 |
+----------+----------+
1 row in set (0.02 sec)

Workflows in the system

MariaDB [wmagent]> SELECT DISTINCT name from wmbs_workflow;
+--------------------------------------------------------------------------------------+
| name                                                                                 |
+--------------------------------------------------------------------------------------+
| pdmvserv_FSQ-RunIISpring15PrePremix-00003_00009_v2_AVE_25_BX_25ns_160309_183500_3378 |
| pdmvserv_SUS-RunIISummer15GS-00153_00279_v0__160316_161202_7826                      |
+--------------------------------------------------------------------------------------+

Files and blocks in Phedex and DBS

Open blocks on DBS:

mysql> SELECT * FROM dbsbuffer_block WHERE status!='Closed';
Empty set (0.04 sec)



Files not updated DBS

mysql> SELECT * from dbsbuffer_file where status = 'NOTUPLOADED';
Empty set (0.069 sec)

Files not injected in Phedex, with parent block id (can be recovered)
MariaDB [wmagent]> SELECT * FROM dbsbuffer_file
WHERE in_phedex=0 
AND block_id IS NOT NULL
AND lfn NOT LIKE '%unmerged%' 
AND lfn NOT LIKE 'MCFakeFile%'
AND lfn NOT LIKE '%BACKFILL%'
AND lfn NOT LIKE '/store/user%';
Empty set (0.70 sec)

Files with no parent (possible input files), cannot be recovered.

MariaDB [wmagent]> SELECT count(*) FROM dbsbuffer_file
WHERE in_phedex=0
AND block_id IS NULL
AND lfn NOT LIKE '%unmerged%' 
AND lfn NOT LIKE 'MCFakeFile%'
AND lfn NOT LIKE '%BACKFILL%'
AND lfn NOT LIKE '/store/user%';
+----------+
| count(*) |
+----------+
|   450723 |
+----------+
1 row in set (0.70 sec)

Running FixPhedex

[cmsdataops@cmsgwms-submit1 current]$ curl https://raw.githubusercontent.com/amaltaro/ProductionTools/master/newFixPhEDEx.py > newFixPhedex.py
[cmsdataops@cmsgwms-submit1 current]$ source /data/srv/wmagent/current/apps/wmagent/etc/profile.d/init.sh
[cmsdataops@cmsgwms-submit1 current]$  python newFixPhedex.py 
100/310 files processed
200/310 files processed
300/310 files processed
Found 24390 out of 24602 files that are already registered in PhEDEx            but buffer doesn't know
Fixing them now, it may take several minutes ...
Rows were successfully updated! Good job!
Starting PhEDExInjector now ...

started with pid 2176643
Done!

Checking the remaining files

MariaDB [wmagent]> SELECT lfn FROM dbsbuffer_file
WHERE in_phedex=0
AND block_id IS NULL
AND lfn NOT LIKE '%unmerged%' 
AND lfn NOT LIKE 'MCFakeFile%'
AND lfn NOT LIKE '%BACKFILL%'
AND lfn NOT LIKE '/store/user%';
Empty set (0.06 sec)


Redeploying



Edit | Attach | Watch | Print version | History: r25 | r19 < r18 < r17 < r16 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r17 - 2016-03-30 - PaolaKatherineRozoBernal
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    CMSPublic All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback