-
Notifications
You must be signed in to change notification settings - Fork 13
JEDI test scripts
This twiki will explain how to submit a JEDI task and drive its status. Please also refer to the instructions on how to submit individual jobs here.
cd /data/atlpan/test/
source /etc/sysconfig/panda_server
source /etc/sysconfig/panda_jedi
#Submit the task according to your current test needs. You can edit the file and change the definition of the tasks.
python addOneTestTaskParamToDEFT.py
python addPileTestTaskParamToDEFT.py
python addAnalTask.py
The task should have been inserted in the DEFT DB
select * from ATLAS_DEFT.t_task order by taskid desc
python -i /opt/panda/lib/python3.6/site-packages/pandajedi/jeditest/taskRefinerTest.py <PRODSOURCELABEL>
You can check the task and dataset were generated in the DB:
SELECT * FROM ATLAS_PANDA.JEDI_Tasks order by jeditaskid desc
SELECT * FROM ATLAS_PANDA.JEDI_DATASETS where jeditaskid = <NEW TASK ID>
The task should be in status defined
python -i /opt/panda/lib/python3.6/site-packages/pandajedi/jeditest/contentsFeederTest.py <PRODSOURCELABEL>
You can check if the files were inserted in the dataset contents table:
SELECT * FROM ATLAS_PANDA.JEDI_DATASET_CONTENTS where jeditaskid = <YOUR TASK ID>
The task status should get updated to assigning for test/managed or ready for user tasks.
(Analysis tasks are not brokered)
python -i /opt/panda/lib/python3.6/site-packages/pandajedi/jeditest/taskBrokerTest.py <PRODSOURCELABEL>
Task should have moved to ready.
If you want to test the job brokerage without generating the jobs (i.e. you can repeat the test multiple times for the same task). This step is not needed if you are going to run the next step with job generation
python -i /opt/panda/lib/python3.6/site-packages/pandajedi/jeditest/jobSplitterTest.py <YOUR TASK ID>
python -i /opt/panda/lib/python3.6/site-packages/pandajedi/jeditest/jobGeneratorTest.py atlas test y WORLD
The jobs can be seen under the jobs tables.
- Jobs are injected via the storeJobs call into pending state (jobswaiting4 table)
- Jobs are moved from pending to assigned/activated state (jobsactive4 table) via the reassignJobs call. This is in fact a PandaClient http call, so the PANDA_URL* env variables need to be set properly for it to work.
SELECT * FROM ATLAS_PANDA.jobswaiting4 where jeditaskid = <YOUR JEDI_TASK_ID>
SELECT * FROM ATLAS_PANDA.jobsactive4 where jeditaskid = <YOUR JEDI_TASK_ID>
Task should have moved to scouting or running.
python finishJob.py <YOUR PANDA_ID>
python add.py
python callbackDDM.py panda.jeditest.log.74ea0d3d-a3f7-4a47-971a-bded4a73da18_sub0106001826 BNL-OSG2_DATADISK
python callbackDDM.py panda.jeditest.EVNT.98e0f3c1-cde8-41e8-9470-11b4410f0dcd_sub0106001825 BNL-OSG2_DATADISK
python add.py
Run the postProcessor to move the task from scouting to running status, once you believe the scouting phase should be done.
python -i /opt/panda/lib/python3.6/site-packages/pandajedi/jeditest/postProcessorTest.py