This directory implements the calibration leg workflow of our NGT demonstrator. It is designed to run the calibration loop on either of our calibration nodes (ngtcalfu-c2b05-43-01.cms and ngtcalfu-c2b05-44-01.cms) for the NGT demonstrator, through monitoring ongoing collisions at CMS and rederiving calibrations (for now only EcalPedestals and SiStrip Bad Components) up to the upload to the conditions database of CMS. Each calibration loop is implemented as a finite state machine (FSM) that monitors input data, processes it, and produces the output for the next step:
- Step 2: Monitors OMS for new runs, processes raw files files from EOS, produces (RE)RECO files
- Step 3: Monitors for step 2 output files, merges them, produces ALCARECO root files
- Step 4: Monitors for step 3 output files, performs harvesting, produces and uploads final payload to condDB to be consumed by the NGT Global Tag
EOS RAW Files
↓
┌─────────────────────────────────────┐
│ Step 2: Raw → RECO │
│ - Monitors OMS for runs │
│ - Processes LS incrementally │
│ - Output: run*_LS*_step2.root │
└─────────────────────────────────────┘
↓ (witness files: *_step2_job.txt)
┌─────────────────────────────────────┐
│ Step 3: RECO → ALCARECO │
│ - Monitors for Step 2 outputs │
│ - Batches files into ALCA jobs │
│ - Output: PromptCalibProd*.root │
└─────────────────────────────────────┘
↓ (witness files: step3_job.txt)
┌─────────────────────────────────────┐
│ Step 4: ALCARECO → DB │
│ - Monitors for Step 3 outputs │
│ - Harvests calibration constants │
│ - Output: {calibration}.db │
└─────────────────────────────────────┘
↓
Automatic Database Upload
The transitions package and the omsAPI folder within the oms-api-client.
All was launched in a tmux session, and all three steps can be run through the sakura user.
Preparing to launch the step 2 script requires:
sudo -u sakura -i
tmux new -s CalibrationLoop2
source /opt/offline/cmsset_default.sh
unset KRB5CCNAME
kinit ${USER}@CERN.CH # in case we are setting up step 2
cmsrel CMSSW_16_0_2
cd CMSSW_16_0_2/src
cmsenv
git clone [email protected]:cms-ngt-hlt/sakura.git
git clone [email protected]:pytransitions/transitions.git
cd transitions
python3 setup.py install --user
cd ../sakura/Calibrations/NGTCalibrationLoop
git clone ssh://[email protected]:7999/cmsoms/oms-api-client.git
cp -r oms-api-client/omsapi .
mkdir -p /tmp/ngt/
chmod g+ws /tmp/ngt/
cp -ru ngtParameters.jsn calibrationYAML/ /tmp/ngt/
python3 NGTLoopStep2.py -c EcalPedestals # or SiStripBadthen quit the tmux session either via keyboard combination or simply closing the terminal.
For step 3:
sudo -u sakura -i
tmux new -s CalibrationLoop3 # make sure to start the tmux session from the sakura account, s.t. the other from the group can also have access to it.
source /opt/offline/cmsset_default.sh
cmsrel CMSSW_16_0_2
cd CMSSW_16_0_2/src
git clone [email protected]:cms-ngt-hlt/sakura.git
git clone [email protected]:pytransitions/transitions.git
cd transitions
python3 setup.py install --user
cmsenv
cd ../sakura/Calibrations/NGTCalibrationLoop
python3 NGTLoopStep3.py -c EcalPedestals # or SiStripBad
tmux detachFor step 4, we simply do:
sudo -u sakura -i
tmux new -s CalibrationLoop4 # make sure to start the tmux session from the sakura account, s.t. the other from the group can also have access to it.
source /opt/offline/cmsset_default.sh
cd CMSSW_16_0_2/src/sakura/Calibrations/NGTCalibrationLoop
cmsenv
python3 NGTLoopStep4.py -c EcalPedestals
tmux detachOne can check what tmux sessions are running and go back to a session through
tmux list-sessinons
tmux attach -t 0
All steps may alternatively be used with -c SiStripBad, these are the only calibration workflows configured for now. They must all be running at the same time to guranatee timely uploads of conditions to the database.
NGTLoopStep2.py continuously queries OMS for collisions run that started within the last 8 hours (which is the time we have to rederive calibrations and the time for which we buffer). Once a suitable run is found, a working directory is created where we start processing the RAW files available on EOS. The jobs are launched cmsDriver.py command. The possible states within this loop are:
- NotRunning - Waiting for a new collisions run to process
- WaitingForLS - Monitoring for new lumi sections (LS)
- CheckingLSForProcess - Evaluating available LS for processing
- PreparingLS - Preparing a batch of LS for job submission
- PreparingFinalLS - Preparing the final batch when run ends
- PreparingExpressJobs - Creating job scripts
- LaunchingExpressJobs - Submitting jobs to process LS
- CleanupState - Finalizing run processing
This step also maintains separate log files for different types of logs --- a complete collection of all can be found in /tmp/ngt/NGTLoopStep2_ALL.log, to monitor activity, one can do tail -f /tmp/ngt/NGTLoopStep2_ALL.log. This step has to be run on a personal cmsusr account due to access needed to EOS.
Step 3 loop processes the output root files of step2 in order to produce the ALCARECO files. The FSM is similar to the one of step 2 described above, used to submit the ALCA jobs. Step 3 can be run on either sakura or personal cmsusr account, it does not really matter here.
Step 4 loop takes all available ALCARECO files available at a given time that were produced from step 3 and performs the ALCAHARVESTING and the eventual upload of the payload to condDB. It re-harvests files as with time we gain more statistics but we still would like to upload conditions payloads as soon as we have them. Step 4 must be run on the sakura account for the eventual upload to the conditions database.
/tmp/ngt/
├── calibrationYAML/
│ ├── SiStripBad.yaml
│ └── EcalPedestals.yaml
├── ngtParameters.jsn
├── NGTLoopStep2_ALL.log # Step 2: All log levels
├── NGTLoopStep2_INFO.log # Step 2: Info only
├── NGTLoopStep2_WARNING.log # Step 2: Warnings only
├── NGTLoopStep2_ERROR.log # Step 2: Errors only
├── NGTLoopStep2_CRITICAL.log # Step 2: Critical only
├── NGTLoopStep3_ALL.log # Step 3: All log levels
├── ...
├── NGTLoopStep4_ALL.log # Step 4: All log levels
├── ...
└── run{run_number}/
├── runStart.log # Created by Step 2: ISO timestamp
├── runEnd.log # Created by Step 2: Signals completion
│
├── CMSSW_X_Y_Z/ # CMSSW release (created by Step 2 jobs)
│
├── cmsDriver_*.sh # Step 2: Job scripts (temporary)
├── run*_LS*_step2.py # Step 2: CMSSW Python configs
├── run*_LS*_step2.log # Step 2: Job logs
├── run*_LS*_step2.root # Step 2: RECO output files
├── run*_LS*_step2_job.txt # Step 2: Witness files
│
├── allLSProcessed.log # Step 2: List of processed LS files
├── expectedOutputs.log # Step 2: Expected Step 2 outputs
├── allStep2FilesProcessed.log # Step 3: List of processed Step 2 files
├── allStep3FilesProcessed.log # Step 4: List of processed Step 3 files
│
├── alcaPromptJob000/ # Step 3: First ALCA job
│ ├── ALCAOUTPUT.sh
│ ├── run*_step3.py
│ ├── stdout.log
│ ├── stderr.log
│ ├── PromptCalibProdEcalPedestals.root
│ └── step3_job.txt # Witness file
│
├── alcaPromptJob001/ # Step 3: Second ALCA job
│ └── ...
│
├── harvestJob000/ # Step 4: First harvesting job
│ ├── HARVESTING.sh
│ ├── run*_step4.py
│ ├── stdout.log
│ ├── stderr.log
│ ├── metadata.json
│ ├── promptCalibConditions.db
│ └── EcalPedestals.db # Final output
│
└── harvestJob001/ # Step 4: Updated harvesting
└── ...
There are quite a lot of issues remaining, still. The version we are at right now is the "functioning" version that was used for the demonstrator in the 2025 data taking. However, for 2026 data-taking, we plan to improve and have worked on all the issues.