Making Ntuples: updated

On LPC slc:

cd nobackup/CMSSW_8_0_30/src

cmsenv

/bin/bash --login

source /cvmfs/cms.cern.ch/crab3/crab.sh

voms-proxy-init -voms cms

DevTools/Utilities/scripts/submit_job.py crabSubmit --site=T3_US_FNALLPC --sampleList qcd.txt --units 300 10May2019_DeepNtuplizer_CA8_qcd DeepNTuples/DeepNtuplizer/production/DeepNtuplizerCA8.py

*****

 

Remember to comment/uncomment lines 6 and 7 in DeepNTuples/DeepNtuplizer/python/QGLikelihood_cfi.py when switching between crab and running locally.

Making Ntuples: old version

Signal samples

Make sure you are not logged in lxplus7. Remember to set up environment on tmux

Remember to comment/uncomment lines 6 and 7 in DeepNTuples/DeepNtuplizer/python/QGLikelihood_cfi.py when switching between crab and running locally.

cd /afs/cern.ch/work/g/ghaza/CMSSW_8_0_30/src 

cmsenv

voms-proxy-init -voms cms 

cmsRun DeepNTuples/DeepNtuplizer/production/DeepNtuplizer.py outputFile=signal_m13_00 inputFiles=/store/mc/RunIISummer16MiniAODv2/SUSYGluGluToHToAA_AToMuMu_AToTauTau_M-13_TuneCUETP8M1_13TeV_madgraph_pythia8/MINIAODSIM/PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/80000/0618D8EF-1AB7-E611-B712-002590FD5838.root

Running over dataset=/SUSYGluGluToHToAA_AToMuMu_AToTauTau_M-*_TuneCUETP8M1_13TeV_madgraph_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

file dataset=/SUSYGluGluToHToAA_AToMuMu_AToTauTau_M-5_TuneCUETP8M1_13TeV_madgraph_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • D767A.root --> signal_m05_00 (41k events)
  • D5694.root --> signal_m05_01 (25k events)
  • 47DDC.root --> signal_m05_02 (30k events)
  • C8F30.root --> signal_m05_03 (28k events) [29 jan 2019]
  • 0FE80.root --> signal_m05_04 (73k events) [29 jan 2019]
  • 13B06.root --> signal_m05_05 (3k events) [29 jan 2019]
  • 97D60.root --> signal_m05_06 (30k events) [29 jan 2019]
  • DC0FB.root --> signal_m05_07 (19k events) [29 jan 2019]
  • A612E.root --> signal_m05_08 (2k events) [29 jan 2019]
  • A6E20.root --> signal_m05_09 (4k events) [29 jan 2019]
  • 98D60.root --> signal_m05_10 (12k events) [29 jan 2019]

file dataset=/SUSYGluGluToHToAA_AToMuMu_AToTauTau_M-9_TuneCUETP8M1_13TeV_madgraph_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • CC860.root --> signal_m09_00 (2k events)
  • 57360.root --> signal_m09_01 (38k events)
  • BB8D.root --> signal_m09_02 (2k events)

file dataset=/SUSYGluGluToHToAA_AToMuMu_AToTauTau_M-11_TuneCUETP8M1_13TeV_madgraph_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • 5F28.root --> signal_m11_00 (74k events)
  • 18C4.root --> signal_m11_01 (69k events)

file dataset=/SUSYGluGluToHToAA_AToMuMu_AToTauTau_M-13_TuneCUETP8M1_13TeV_madgraph_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • 5838.root --> signal_m13_00 (73k events)
  • 8BA2.root --> signal_m13_01 (66k events)
  • 2A40.root --> signal_m13_02 (77k events)

file dataset=/SUSYGluGluToHToAA_AToMuMu_AToTauTau_M-15_TuneCUETP8M1_13TeV_madgraph_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • C3A46.root -> signal_m15_00 (38k events)
  • E5BE.root --> signal_m15_01 (33k events)
  • A608C.root --> signal_m15_02 (68k events)

file dataset=/SUSYGluGluToHToAA_AToMuMu_AToTauTau_M-17_TuneCUETP8M1_13TeV_madgraph_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • B559.root --> signal_m17_00 (78k events)
  • FE80.root --> signal_m17_02 (13k events)
  • 45ED.root --> signal_m17_03 (20k events)

file dataset=/SUSYGluGluToHToAA_AToMuMu_AToTauTau_M-19_TuneCUETP8M1_13TeV_madgraph_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • 855C.root -> signal_m19_00 (160k events)
  • 0C52.root --> signal_m19_01(144k events)

5 OCTOBER:


cmsRun DeepNTuples/DeepNtuplizer/production/DeepNtuplizer.py outputFile=5oct_signalm17_00 inputFiles=/store/mc/RunIISummer16MiniAODv2/SUSYGluGluToHToAA_AToMuMu_AToTauTau_M-17_TuneCUETP8M1_13TeV_madgraph_pythia8/MINIAODSIM/PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/60000/96002A2A-09CE-E611-9873-14DDA9D4F168.root 

Background samples

QCD

Since this is taking a while, first I will run over a sample in each pt distribution. Then I will go back if necessary and run over more files from the same datasets.

file dataset=/QCD_Pt_15to30_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

*609A.root --> QCDPt15to30_00 (168k events)

Full file name: /store/mc/RunIISummer16MiniAODv2/QCD_Pt_15to30_TuneCUETP8M1_13TeV_pythia8/MINIAODSIM/PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/70000/00BD0FD2-FCB2-E611-BA46-0025905A609A.root

dataset=/QCD_Pt_30to50_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

file dataset=/QCD_Pt_30to50_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • 766C.root --> QCDPt30to50_00_0.root (149k events)

dataset=/QCD_Pt_50to80_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • BA4C.root --> segmentation violation
  • BD8D4.root --> QCDPt50to80_00_0.root (147k events)
  • 34E8B.root --> QCDPt50to80_01 (99k events)

dataset=/QCD_Pt_80to120_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

/store/mc/RunIISummer16MiniAODv2/QCD_Pt_80to120_TuneCUETP8M1_13TeV_pythia8/MINIAODSIM/PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/70000/F4C8D13D-07B7-E611-888D-A0000420FE80.root

  • FE80.root -->QCDPt80to120_00_0.root (133k events)
  • 74C8.root --> QCDPt80to120_01 (56k events)
  • 974A0.root --> QCDPt80to120_02 (70k events)

dataset=/QCD_Pt_120to170_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • 4284.root --> QCDPt120to170_00.root (116k events)
  • 8BA4C.root --> QCDPt120to170_01 (76k events)

dataset=/QCD_Pt_170to300_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • 5B8568.root -> QCDPt170to300_ (74k events)
  • 5C42A4.root --> QCDPt170to300_01 (103k events)

dataset=/QCD_Pt_300to470_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

file dataset=/QCD_Pt_300to470_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • C89.root --> QCDPt300to470_00_0.root (53k events)
  • D768E.root --> QCDPt300to470_01 (10k events)
  • 5298.root --> QCDPt300to470_02 (69k events)
  • C7D8F.root --> QCDPt300to470_03 (50k events)

file dataset=/QCD_Pt_470to600_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • C7D4.root --> QCDPt470to600_00 (92k events)
  • 6E64.root --> QCDPt470to600_01 (90k events)

file dataset=/QCD_Pt_600to800_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • 3504DA.root --> QCDPt600to800_00 (85k events)
  • 7D13E.root --> QCDPt600to800_01 (87k events)

file dataset=/QCD_Pt_800to1000_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • D0679.root --> QCDPt800to1000_00 (89k events)
  • D762A.root --> QCDPt800to1000_01 (88k events)

file dataset=/QCD_Pt_1000to1400_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • D08D6.root --> QCDPt1000to1400_00 (80k events)
  • 8D74.root --> QCDPt1000to1400_01 (70k events)

file dataset=/QCD_Pt_1400to1800_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • 48E4.root --> QCDPt1400to1800_00 (84k events)
  • 60E0.root --> QCDPt1400to1800_01(84k events)

file dataset=/QCD_Pt_1800to2400_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • 9070.root --> QCDPt1800to2400_00 (78k events)
  • 908C.root --> QCDPt1800to2400_01 (50k events)
  • 263D.root --> QCDPt1800to2400_02 (39k events)

file dataset=/QCD_Pt_2400to3200_TuneCUETP8M1_13TeV_pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

  • 7606.root --> QCDPt2400to3200_00 (69k events)
  • C3434.root --> QCDPt2400to3200_01 (41k events)
  • D7604.root --> QCDPt2400to3200_02 (44k events)

TTbar

dataset=/TT_TuneCUETP8M2T4_13TeV-powheg-pythia8/RunIISummer16MiniAODv2-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/MINIAODSIM

/store/mc/RunIISummer16MiniAODv2/TT_TuneCUETP8M2T4_13TeV-powheg-pythia8/MINIAODSIM/PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/50000/0693E0E7-97BE-E611-B32F-0CC47A78A3D8.root

  • C35A4.root--> tt_00 (95k events)
  • DD98.root--> tt_01_0.root (80k events)
  • F7BA.root --> tt_02_00.root (94k events)
  • 4C669C.root --> tt_03 (53k events)
  • D767E.root --> tt_04 (89k events)
  • 12D4F.root --> tt_05 (89k events)
  • 76A0.root --> tt_06 (90k events)
  • FE80.root --> tt_07 (93k events)
  • 6132.root --> tt_08 (85k events)
  • A64C.root --> tt_09 (91k events)

Because there are over 900 files in the dataset, I list the full file names since it is not easy to find them from the list only:

/store/mc/RunIISummer16MiniAODv2/TT_TuneCUETP8M2T4_13TeV-powheg-pythia8/MINIAODSIM/PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/80000/FC87666B-ADBE-E611-8AF6-0CC47A7C35A4.root

/store/mc/RunIISummer16MiniAODv2/TT_TuneCUETP8M2T4_13TeV-powheg-pythia8/MINIAODSIM/PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/80000/FAF44DD3-A2BE-E611-8B89-002590E7DD98.root

/store/mc/RunIISummer16MiniAODv2/TT_TuneCUETP8M2T4_13TeV-powheg-pythia8/MINIAODSIM/PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/80000/FA6FA157-A0BE-E611-A9F0-001E67E6F7BA.root

/store/mc/RunIISummer16MiniAODv2/TT_TuneCUETP8M2T4_13TeV-powheg-pythia8/MINIAODSIM/PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/80000/FE588D8C-ABBE-E611-B08E-0025904C669C.root

/store/mc/RunIISummer16MiniAODv2/TT_TuneCUETP8M2T4_13TeV-powheg-pythia8/MINIAODSIM/PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/80000/FCBCFDB5-AFBE-E611-8462-0CC47A4D767E.root

/store/mc/RunIISummer16MiniAODv2/TT_TuneCUETP8M2T4_13TeV-powheg-pythia8/MINIAODSIM/PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/80000/FA66429B-9CBE-E611-8499-02163E012D4F.root

/store/mc/RunIISummer16MiniAODv2/TT_TuneCUETP8M2T4_13TeV-powheg-pythia8/MINIAODSIM/PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/80000/FA5C0E3C-A8BE-E611-92B5-0CC47A4D76A0.root

/store/mc/RunIISummer16MiniAODv2/TT_TuneCUETP8M2T4_13TeV-powheg-pythia8/MINIAODSIM/PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/80000/FA5913D2-9CBE-E611-81CB-A0000420FE80.root

/store/mc/RunIISummer16MiniAODv2/TT_TuneCUETP8M2T4_13TeV-powheg-pythia8/MINIAODSIM/PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/80000/F62D9F7A-A2BE-E611-A346-0025905A6132.root

/store/mc/RunIISummer16MiniAODv2/TT_TuneCUETP8M2T4_13TeV-powheg-pythia8/MINIAODSIM/PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1/80000/F4F73F4A-A1BE-E611-BA58-549F3525A64C.root

Converting from NTuple to python object

Need to log into lxplus7

 

cd /afs/cern.ch/work/g/ghaza/DeepJets/DeepJetditau/environment

source lxplus_env.sh / gpu_env.sh 

cd ../modules

python ../convertFromRoot/convertFromRoot.py -i ../../../CMSSW_8_0_30/src/list_tt_00.txt -o ../outputconvertr2py_name -c TrainData_diTau

18 JUL iteration:


python ../convertFromRoot/convertFromRoot.py -i ../../../CMSSW_8_0_30/src/biglist00.txt -o ../outputconvertr2py_biglist18jul00 -c TrainData_diTau

Where biglist00.txt is

signal_m05_00_0.root

signal_m05_01_0.root

signal_m09_00_0.root

signal_m09_01_0.root

signal_m09_02_0.root

signal_m11_00_0.root

signal_m11_01_0.root

signal_m13_00_0.root

signal_m15_00_0.root

signal_m15_01_0.root

signal_m19_00_0.root

signal_m19_01_0.root

QCDPt30to50_00_0.root

QCDPt50to80_00_0.root

QCDPt80to120_01_0.root

QCDPt120to170_01_0.root

tt_00_0.root

tt_02_0.root

tt_03_0.root

tt_06_0.root

tt_07_0.root

5 OCTOBER:

python ../convertFromRoot/convertFromRoot.py -i ../../../CMSSW_8_0_30/src/5octbiglist.txt -o ../outputconvertr2py_biglist5oct -c TrainData_diTau

16 OCT:

../convertFromRoot/convertFromRoot.py -i ../../../CMSSW_8_0_30/src/list_tt_00.txt -o ../outputconvertr2py_tt_00_16_oct -c TrainData_diTau

29 JAN:

python ../convertFromRoot/convertFromRoot.py -i ../../../CMSSW_8_0_30/src/list_signal_m05_04.txt -o ../outputconvertr2py_signal_m05_04 -c TrainData_diTau

Training

example of command line input:


cd /afs/cern.ch/work/g/ghaza/DeepJets/DeepJetditau/Train

python diTau.py ../outputconvertr2pyQCD30to50_0/dataCollection.dc ../train_QCDPt30to50_output_03

8 JUL iteration:


python diTau.py ../outputconvertr2py_biglist18jul00/dataCollection.dc ../train_biglist18jul00

5 OCTOBER:

python diTau.py ../outputconvertr2py_biglist5oct/dataCollection.dc ../train_biglist5oct (found empty, corrupted)

16 OCTOBER:


python diTau.py ../outputconvertr2py_tt_00_16_oct/dataCollection.dc ../train_tt_00_16oct

python diTau.py ../outputconvertr2py_tt_00_16_oct/dataCollection.dc ../train_tt_00_16octscreenv3

python diTau.py ../outputconvertr2pyditau01/dataCollection.dc ../ditau01_29jan

29 JAN:


python diTau.py ../outputconvertr2py_signal_m05_04/dataCollection.dc ../train_signal_m05_04

Converting test data

Make list in CMSSW_8_0_30/src (text file with name of probably only one root file in same directory)


cd /afs/cern.ch/work/g/ghaza/DeepJets/DeepJetditau/convertFromRoot

 ./convertFromRoot.py --testdatafor ../train_13jul00output/trainsamples.dc -i ../../../CMSSW_8_0_30/src/test13julsignalm05_02.txt -o ../testdatasignalm05_02convertoutput

18 JUL iteration:


./convertFromRoot.py --testdatafor ../train_biglist18jul00/trainsamples.dc -i ../../../CMSSW_8_0_30/src/test13julsignalm05_02.txt -o ../testdatasignalm05_02_biglist18jul00convertoutput

./convertFromRoot.py --testdatafor ../train_biglist18jul00/trainsamples.dc -i ../../../CMSSW_8_0_30/src/test_QCDPt50to80_01_18july.txt -o ../testdataQCDPt50to80_01convertoutput

./convertFromRoot.py --testdatafor ../train_biglist18jul00/trainsamples.dc -i ../../../CMSSW_8_0_30/src/test_tt_08_18july.txt -o ../testdatatt_08convertoutput

16 OCT:

 
./convertFromRoot.py --testdatafor ../train_tt_00_16octscreenv3/trainsamples.dc -i ../../../CMSSW_8_0_30/src/test_tt_08_18july.txt -o ../test_tt_08_16octconvertoutput

29 Jan:


./convertFromRoot.py --testdatafor ../train_signal_m05_04/trainsamples.dc -i ../../../CMSSW_8_0_30/src/list_signal_m05_07.txt -o ../testdata_signal_m05_07_convertoutput

Applying test data


 /afs/cern.ch/work/g/ghaza/DeepJets/DeepJetditau

scripts/predict.py train_13jul00output/KERAS_model.h5 outputconvertr2py_12jul01/dataCollection.dc modelapplied_12jul00

18 JULY iteration:


scripts/predict.py train_biglist18jul00/KERAS_model.h5 testdatasignalm05_02_biglist18jul00convertoutput/dataCollection.dc modelapplied_signal_m05_02_biglist_18july00 --labels

scripts/predict.py train_biglist18jul00/KERAS_model.h5 testdataQCDPt50to80_01convertoutput/dataCollection.dc modelapplied_QCDPt50to80_01_biglist_18july00 --labels

scripts/predict.py train_biglist18jul00/KERAS_model.h5 testdatatt_08convertoutput/dataCollection.dc modelapplied_tt_08_biglist_18july00 --labels

29 JAN:


scripts/predict.py train_signal_m05_04/KERAS_model.h5 testdata_signal_m05_07_convertoutput/dataCollection.dc modelapplied_signal_m05_04_to_signal_m05_07 --labels

output: created predition friend tree /afs/cern.ch/work/g/ghaza/DeepJets/DeepJetditau/modelapplied_signal_m05_04_to_signal_m05_07/signal_m05_07_0_predict.root for /afs/cern.ch/work/g/ghaza/CMSSW_8_0_30/src/signal_m05_07_0.root

Setting up for GPUs on LPC

if not in bash:

 /bin/bash --login

Once you have miniconda (from https://conda.io/en/latest/miniconda.html)


cd miniconda
source bin/activate
cd DeepJet
cd environment
conda create --copy --name deepjetLinux3 python=2.7.5
conda install --name deepjetLinux3 --file deepjetLinux3.conda

source activate deepjetLinux3

conda install pip

pip uninstall numpy
conda uninstall numpy
conda install root-numpy

Set up specific for GPU nodes:

*working on this now, redoing above but replace "deepjetLinux3" with "deepjetLinux3GPU"

*Remember to use option -n 6 for convert from root on SL7. This uses fewer threads (courteous to other computer users)

We will use SLC 7 for converting to root over ntuples. Use SLC 7 GPU nodes for training.

export CUDA_HOME=/cvmfs/cms-lpc.opensciencegrid.org/sl7/usr/local/cuda
export PATH=$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/extras/CUPTI/lib64:$CUDA_HOME/lib64:$LD_LIBRARY_PATH

conda install cudnn

Logging on for GPUs on LPC

17 Feb Ntuplizer step (didn't actually need to do this today)

cd  /uscms_data/d3/ghaza/CMSSW_8_0_30/src

cmsenv

make list of datasets (?) 17Feb2019_00.txt

make sure you are in bash shell

source /cvmfs/cms.cern.ch/crab3/crab.sh

DevTools/Utilities/scripts/submit_job.py crabSubmit --site=T3_US_FNALLPC --sampleList signal.txt 20Feb2019_DeepNtuplizerAK8_signal DeepNTuples/DeepNtuplizer/production/DeepNtuplizerAK8.py 

output will be in /eos/uscms/store/user/ghaza (not accessible on GPU nodes)

or it will be in /uscms_data/d3/ghaza/crab_projects

Converting from root

make list of samples located on /eos/uscms/store/user/ghaza area in src directory of relevant CMSSSW (remember to split a training and testing sample) and changes lines 11 and 12 in /uscms/home/ghaza/nobackup/CMSSW_8_0_30/src/DevTools/Utilities/scripts/copytoScratch.sh and run this script ON the GPU nodes

bash DevTools/Utilities/scripts/copytoScratch.sh

now make a list of samples using their new /storage/local/data1/gpuscratch/ghaza address, put this text file in the src directory of the CMSSW

emacs -nw 17Feb2019_00_gpuscratch.txt

*It might be necessary to log out and log back in to set up this environment

cd miniconda
source bin activate

cd DeepJet/environment
source lpc_gpu_env.sh

cd ../modules

python ../convertFromRoot/convertFromRoot.py -n 6 -i ../../../CMSSW_8_0_30/src/17Feb2019_00_gpuscratch.txt -o /uscms/home/ghaza/nobackup/miniconda/DeepJet/convertFromRootoutput/17Feb2019 -c TrainData_diTau

Training

cd ../Train

python diTau.py ../convertFromRootoutput/17Feb2019/dataCollection.dc ../train17Feb2019

-- GraceMarieHaza - 2018-06-27

Edit | Attach | Watch | Print version | History: r37 < r36 < r35 < r34 < r33 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r37 - 2019-05-17 - GraceMarieHaza
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Sandbox All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback