Running simulation jobs with PBS scheduler

This page shows us how to run simulation jobs on a cluster with a PBS job scheduler. These example PBS script files are bundled together with Lumerical on Linux.

Submission script

1. The template for a PBS job scheduler will probably be similar as shown here.

   #PBS -S /bin/bash
   #PBS -l mem=<total_memory>mb
   #PBS -l walltime=<hours>:<minutes>:<seconds>
   #PBS -l software=<product_license>
   
   cd $PBS_O_WORKDIR
   module load <your_application>
   echo "Starting run at: 'date'"
       <application_command_line>
   echo "Job finished at: 'date'"
   exit

2. The information inside <> signs must be provided according to the application you want to run.
3. Update this with the appropriate information for your first FDTD simulation job submission.
4. We recommend using a small simulation file for your tests.
5. The updated template should look like below,

   #PBS -S /bin/bash
   #PBS -l mem=100mb
   #PBS -l walltime=1:00:00
   #PBS -l software=FDTD
   
   cd $PBS_O_WORKDIR
   module load lumerical
   module load <openmpi>
   echo "Starting run at: 'date'"
       mpiexec -n 4 /opt/lumerical/[[verpath]]/bin/fdtd-engine-ompi-lcl ./path/simulation.fsp
   echo "Job finished at: 'date'"
   exit

6. The 'application_command_line' (the one that executes the simulation) depends on the MPI you will be using.
7. To submit it in a PBS system use 'qsub' and the name of the submission script,

   qsub <submission_script>.sh

Submitting multiple simulations

Having tested the process to submit one simulation file, you can now automate the process for multiple simulation files. For this purpose, we provide a set of three shell script files for a Linux based PBS scheduling system, which can be easily reused and customized:

1. Master script

   /opt/lumerical/[[verpath]]/bin/fdtd-run-pbs.sh

   • This script might require some minor customization.
   • It takes the number of processes per job (optional, default is 8 processes) and the fsp files as arguments.
   • It then generates a job submission script for each simulation file using the PBS template script and the simulation information provided by the process template script.
   • Finally, the script submits the jobs using qsub.
2. PBS template script

   /opt/lumerical/[[verpath]]/templates/fdtd-pbs-template.sh

   • This is the template script that is used to create the job submission and it must be customized to ensure that the submission script follows the requirements from your system administrator.
   • The modifications are essentially the same as the ones described above - "Submission script".
3. Process template script

   /opt/lumerical/[[verpath]]/bin/fdtd-process-template.sh

   • This script rarely needs customization.
   • It replaces the tags in the PBS template script for;
     • total memory required for all processes <total_memory>
     • memory required by each process <processor_memory>
     • total time for the simulation <hours>,<minutes>,<seconds>
     • number of processes to use <n>
     • name of the fsp project file <filename>
   • The file name and the number of processes are provided when calling the master script.
   • For all the other tags, default values will be used unless the appropriate options are provided when calling the process template script from the master script.
4. Once the shell script files are configured, you can submit jobs to the scheduler,

   sh /opt/lumerical/[[verpath]]/bin/fdtd-run-pbs.sh -n <procs> fsp1 fsp2 ... [fspN]

5. For example, to submit all the fsp files in a directory to the scheduler, you can use,

   sh /opt/lumerical/[[verpath]]/bin/fdtd-run-pbs.sh *.fsp

Script customization

1. The script files are documented with comments in case they need to be customized.
2. The process template script should not be modified unless some advanced customization is required.
3. If you modify the master and/or PBS template scripts, we recommend that you make a copy of them under a different name so that your changes will not be overwritten when you upgrade the software. e.g. the master script can be renamed to 'myfdtd-run.sh'.
4. If you change the name of the PBS template script, you must modify the TEMPLATE variable in the master script to specify the correct template file.

See also

Job Scheduler Submission Scripts (SGE, Slurm, Torque) 

Lumerical Job Scheduler Integration Configuration (Slurm, Torque, LSF, SGE)