Overview
This article outlines the process of running a large simulation job across several computers on a local network or cluster that requires more memory that is not available on a single computer or node on your cluster.
Important
- This feature is currently supported and available for FDTD and varFDTD in MODE.
- Machines should be the same operating system and the same version of Lumerical.
- Configure your machines/cluster as shown in the Configuring your cluster for Ansys Lumerical page.
- Before trying to distribute a single simulation across multiple computers, ensure that simulations run properly on each computer individually.
- On a local network, this might not speed up your simulation due to the local network's bandwidth.
Setting the Resource Configuration
Windows
- Create or open your simulation project in the Lumerical CAD/GUI
- Open "Resources"
- Add the hosts' name or IP of the machines that will be running the simulation separate by a ",".
- Set the number of MPI processes to be used per host, separated by a ",".
- Run each MPI process on 1 thread/core and 1 instance/capacity on each machine.
- "Edit" the resource.
- Use "Remote: Intel MPI" as the job launching preset as shown here.
- Select "Custom" for custom MPI installations. e.g. clusters or HPC that have a preconfigured MPI implementation.
- "Run Test" to check your settings.
- Save and close your resource configuration.
Linux
- Install Intel MPI on the machines that will be running your simulation.
- Open Terminal and set Intel MPI environment.
/opt/intel/impi/2019.9.304/intel64/bin/mpivars.sh
- Create or open your simulation project in the Lumerical CAD/GUI
- Open "Resources"
- Change the Name of the resource for reference.
- "Edit" the resource.
- Select "Custom" as the job launching preset.
- Add the Intel MPI and FDTD executables accordingly.
- Enter the total number of MPI processes(-n), ranks per node (-ppn) and the hostnames/IP (-hosts, comma separated) that will be running the simulation in the "extra command line options", under "MPI options".
- Check/Enable "no default options".
- "Run Test" to check your settings.
- Save your settings.
Run directly from the Terminal
- Open Terminal.
- Change directory to the location of your simulation file.
- Set Intel MPI environment.
- Run using Intel MPI runtime executable with the corresponding MPI processes (-n) and ranks per node (-ppn) and remote machines (-hosts) that will be running the simulation.
cd /path/to/simulationfile/
/opt/intel/impi/2019.9.304/intel64/bin/mpivars.sh
/opt/intel/impi/2019.9.304/intel64/bin/mpirun -n 16 -ppn 4 -hosts localhost,host2,host3,host4 /opt/lumerical/[[verpath]]/bin/fdtd-engine-impi-lcl -t 1 -logall simulationfile.fsp
- Create a machinefile containing the host's names and corresponding ranks per hosts.
- Run the simulation with the total number of MPI processes (-n) after the machine file option.
#machinefile content
cat machinefile.txt
localhost:4
remote2:8
node3:4
#run simulation using Intel with -machinefile option
#number of processes comes after the machinefile
/opt/intel/impi/2019.9.304/intel64/bin/mpirun -machinefile -machinefile.txt -n 16 /opt/lumerical/[[verpath]]/bin/fdtd-engine-impi-lcl -t 1 -logall simulationfile.fsp
See Controlling Process Placement with the Intel® MPI Library for more information.
Calculating how many solver licenses are required
The number of solver/accelerator/HPC licenses required to run your jobs depends on the number of cores used, the number of concurrent simulations, the type of job, and the type of license you have purchased. See the Understanding solver, accelerator, and HPC license consumption page for details.
See also
Compute resource configuration use cases – Ansys Optics
Resource configuration elements and controls – Ansys Optics
Lumerical solve, accelerator and Ansys HPC license consumption – Ansys Optics
Configuring resources for parallel jobs across several computers on Windows – Ansys Optics