Overview
This article outlines the process of running a large simulation job across several computers on a local network or cluster that requires more memory that is not available on a single computer or node on your cluster.
Important
- This feature is currently supported and available for FDTD and varFDTD in MODE.
- Machines should be the same operating system and the same version of Lumerical.
- Configure your machines/cluster as shown in the Configuring your cluster for Ansys Lumerical page.
- Before trying to distribute a single simulation across multiple computers, ensure that simulations run properly on each computer individually.
- On a local network, this might not speed up your simulation due to the local network's bandwidth.
Setting the Resource Configuration
Windows
- Create or open your simulation project in the Lumerical CAD/GUI
- Open "Resources"
- Add the hosts' name or IP of the machines that will be running the simulation separate by a ",".
- Set the number of MPI processes to be used per host, separated by a ",".
- Run each MPI process on 1 thread/core and 1 instance/capacity on each machine.
- "Edit" the resource.
- Use "Remote: Intel MPI" as the job launching preset as shown here.
- Select "Custom" for custom MPI installations. e.g. clusters or HPC that have a preconfigured MPI implementation.
- "Run Test" to check your settings.
- Save and close your resource configuration.
Linux
- Install Intel MPI.
- Set Intel MPI environment.
/opt/intel/impi/2019.9.304/intel64/bin/mpivars.sh
- Create or open your simulation project in the Lumerical CAD/GUI
- Open "Resources"
- Change the Name of the resource for reference.
- "Edit" the resource.
- Select "Custom" as the job launching preset.
- Enter the Intel MPI and FDTD executable in the "executable" fields.
- Check/Enable "no default options".
- Enter the total number of MPI processes (-n), ranks per node (-ppn) and the hostnames/IP (-hosts, comma separated) of the computers that will be running the simulation in the "extra command line options", under "MPI options".
- "Run Tests".
- Save and apply settings.
Run using a 'machinefile'
- Create a 'machinefile' containing the host names of the computers that will be running the simulation and corresponding ranks per host separated by a ":" (colon).
#example of host/machine file contents
localhost:4
remote2:8
node3:4
- Follow Steps 1 to 7 above.
- Enter the "-machine" argument followed by the machinefile's filename in the "extra command line options", under "MPI options".
- Run Tests.
- Save and apply settings.
Run from Terminal
- Open Terminal.
- Change directory to the location of your simulation file.
- Set Intel MPI environment.
- Run using Intel MPI runtime executable with the corresponding MPI processes (-n) and ranks per node (-ppn) and remote machines (-hosts) that will be running the simulation and the FDTD Intel MPI engine executable.
cd /path/to/simulationfile/
/opt/intel/impi/2019.9.304/intel64/bin/mpivars.sh
mpirun -n 16 -ppn 4 -hosts localhost,host2,host3,host4 /opt/lumerical/[[verpath]]/bin/fdtd-engine-impi-lcl -t 1 simulationfile.fsp
Using a machine file
- When using a machine file, enter the "-machine" argument followed by the machinefile's filename and the FDTD Intel MPI engine executable.
cd /path/to/simulationfile/
/opt/intel/impi/2019.9.304/intel64/bin/mpivars.sh
mpirun -machinefile /path/machinefile.txt /opt/lumerical/[[verpath]]/bin/fdtd-engine-impi-lcl -t 1 simulationfile.fsp
See Controlling Process Placement with the Intel® MPI Library for more information.
Calculating how many solver licenses are required
The number of solver/accelerator/HPC licenses required to run your jobs depends on the number of cores used, the number of concurrent simulations, the type of job, and the type of license you have purchased. See the Understanding solver, accelerator, and HPC license consumption page for details.
See also
Compute resource configuration use cases – Ansys Optics
Resource configuration elements and controls – Ansys Optics
Lumerical solve, accelerator and Ansys HPC license consumption – Ansys Optics
Configuring resources for parallel jobs across several computers on Windows – Ansys Optics