LSF job submission at CCJ

Modification of job submission method
The job submission method for LSF in CCJ has been slightly modified from original LSF command to optimize I/O load balance in each computing node. Basic LSF commands are same as before e.g. bqueues, bjobs and so on. The only difference is some options at the job submission, bsub, and the location of your working directory in each node i.e. job_tmp. Here is the modification.


Available options:
• -q [queue name, short/long/big]
• -s [your script]
• -o [file name for standard output]
• -e [file name for standard error]
• -m [node name, ap???]
• -r [run number of PHENIX data]
• -J [job name]

In particular, -s option is necessary to specify your script to be run in each node, and put your script and arguments into double quotation as follows.



bsub -q short -o stdout.txt -e stderr.txt -s "YourScript Arguments ..."


The working directories in each node are dynamically changed job-by-job. You can get the location of the working directory, which is conventional job_tmp, by an environmental variable in your script. If you use csh/bash, the variable of


$CCJ_JOBTMP

indicates the correct job_tmp directory at the running node. In the case of you use the perl script, you can get the location as follows.


$ENV{CCJ_JOBTMP}

For nDST analyzer
Actually, many types of nDSTs have already been staged at the local disks in each node. If you want to do the nDST based analysis, please specify the run number e.g 123456 with -r option at the job submission as.



bsub -q short -o stdout.txt -e stderr.txt -r 123456 -s "YourScript Arguments ..."


The location of nDST in each node will be assigned to an environmental variable of


$CCJ_DATADIR

in the same way as $CCJ_JOBTMP. You can drastically save the time for the transfer of input data from NFS storages to each node.
If you want to know the file name of each nDST, a command have been prepared to seek the corresponding file name with specified run number. For example,



/opt/ccj/bin/filename 259051


and the result is



DST_EVE_ALL_run8pp_200GeV_pro82-0000259051-9000.root DST_EVE_ALL_run8pp_200GeV_pro82-0000259051-9001.root DST_EVE_ALL_run8pp_200GeV_pro82-0000259051-9002.root DST_MPC_ERT_run8pp_200GeV_pro82-0000259051-9000.root DST_MPC_ERT_run8pp_200GeV_pro82-0000259051-9001.root DST_MPC_ERT_run8pp_200GeV_pro82-0000259051-9002.root DST_MPC_MB_run8pp_200GeV_pro82-0000259051-9000.root DST_MPC_MPC_run8pp_200GeV_pro82-0000259051-9000.root DST_MPC_MT_run8pp_200GeV_pro82-0000259051-9000.root DST_MPC_OT_run8pp_200GeV_pro82-0000259051-9000.root generic_muid_effic_ntuple_run8pp_200GeV_pro82-0000259051.tar hist_run8pp_200GeV_pro82-0000259051.tar MWG_ERT_run8pp_200GeV_pro82-0000259051-9000.root MWG_MB_run8pp_200GeV_pro82-0000259051-9000.root MWG_MT_run8pp_200GeV_pro82-0000259051-9000.root MWG_MU_run8pp_200GeV_pro82-0000259051-9000.root MWG_OT_run8pp_200GeV_pro82-0000259051-9000.root


You can find some lists of available run numbers, which have already been staged at the local disks in each node. Here is the sample nDSTs for your code development. It is accessible from interactive nodes (i.e. linux2/3/4/5).


/ccj/w/r03/Sample_nDST/

Followings are examples of some simple job submission scripts.


• In case you want to get the file size of the specific type of nDST.
  
  

  #!/bin/csh -f ### Script name: sub.csh set runlist = (276531 276532 276533) @ i = 1 while ($i <=$#runlist )   bsub -q short -o $runlist[$i].out -e $runlist[$i].err -r $runlist[$i] -s "run.csh $runlist[$i] CNT_ERT"   @ i ++ end

  
  

  #!/bin/csh -f ### Script name: run.csh set RunNumber = $1 set NdstType = $2 set output = file_size_$RunNumber_$NdstType.dat ### Firstly, move to the working directory in each node. cd $CCJ_JOBTMP ### Getting file size. foreach list (`/opt/ccj/bin/filename $RunNumber | grep $NdstType`)   ls -l $CCJ_DATADIR/$list >> ./$output end ### Get back the result to your working directory on NFS server. /opt/ccj/bin/rcpx ./$output ccjnfs20:/ccj/w/r01/...

  
  It is important to specify the server name as in red font (ccjnfs20:) before the direcotry on NFS server. You can confirm the name of correspondfing NFS server at the interactive node as follows e.g. data51.
  
  

  [linux5]$ ls /ccj/w/data51 [linux5]$ df