Evaluate for Free

Distributed computing

Overview

This article outlines the process of running a large simulation job across several computers on a local network or cluster that requires more memory that is not available on a single computer or node on your cluster. 

distributed_computing_overview.png

Important

  • MPI setup is required.
  • This feature is currently supported and available for FDTD, varFDTD, and EME in MODE starting with the 2024 R1 release.
    • Distributed computing with GPU solver on FDTD is supported with all MPIs, but a CUDA-aware MPI is recommended for maximum performance.  For more information on using GPU for FDTD simulations, see the Getting started with running FDTD on GPU Knowledge Base article.
  • Machines should be the same operating system and the same version of Lumerical.
  • Machines in your cluster should be configured according to this Knowledge Base article.
  • Ensure that simulations run on each computer individually.
  • On a "local office network", distributed computing might not speed up your simulation due to limitations in the network's bandwidth.

Setting up MPI

  • Windows: Intel MPI is installed with the product and can be used for distributed computing. Configure the Intel MPI using the instructions from this Knowledge Base article. The Microsoft MPI cannot be used for distributed computing.

    Note: See Knowledge Base articles for automated installer and standalone installer instructions on installing the Intel MPI when a version of Visual Studio is already installed.
  • Linux: Intel MPI and Open MPI can be used for distributed computing. Install and configure the MPI using instructions from this Knowledge Base article.

Note: On older versions of the Lumerical software and Intel MPI, additional configuration may be required. See the section “Workarounds on older versions” in this Knowledge Base article for these configurations.

Running from the CAD/IDE

Using Intel MPI or Open MPI: comma-separated list

For distributed computing, a comma-separated list for the name of the nodes can be entered, as shown below.

Picture11.png

When using Intel MPI, only one value is allowed to be entered in the “Processes” column, and that value corresponds to the number of processes to be executed on each node.

When using Open MPI, a comma-separated list is allowed for the “Processes” column, corresponding to the number of slots on each node corresponding to the node list. If a single number is inputted, each node will use the same assigned number of process or slot.

Picture12.png

This configuration can be used for both CPU and GPU simulations. For GPU simulations, additional setup for the MPI and other commands may be required. See the Knowledge Base article on getting started with FDTD on GPU for more details.

Using Intel MPI: advanced settings (Linux and Windows)

Resource configuration

To configure a distributed computing resource using Intel MPI, first follow the instructions above on setting up your cluster and the MPI, then follow these steps:

  1. Create or open your simulation project in the Lumerical CAD/GUI.
  2. Open "Resources".
  3. Change the Name of the resource for reference.
  4. "Edit" the resource.
  5. Select "Remote: Intel MPI" (2024 and newer) or "Custom" (2023 R2 and older) as the job launching preset. 
  6. Enter the MPI and FDTD executable in their corresponding fields. 
  7. Check "no default options".  
  8. Enter the total number of MPI processes (-n), processes per node (-ppn), and the hostnames/IP (-hosts, comma separated) of the computers, in the "extra command line options", under "MPI  options".
  9. Save to apply your changes. 

The resource configuration for distributed computing is now complete, and you can proceed to running the simulation. For more information on command line flags to set processes per node, see Intel MPI documentation on Controlling Process Placement with the Intel® MPI Library.

An example is resource configuration advanced options window shown in the figure below for Windows.

Picture13.png

Note: "Run Tests" will not work with the manual MPI settings (no default options).

Using a machine file

As an alternative to using the command line arguments for -hosts and -ppn, you can also use a machine file containing the host names of the computer and corresponding processes per node separated by a “:” (colon).

An example is shown below:

#example of host/machine file contents
localhost:4
remote2:8
node3:4

To use this file through the CAD window, follow the steps below:

  1. Follow Steps 1 to 7 above. 
  2. Enter the "-machine" argument followed by the machine file's filename in the "extra command line options", under "MPI options".
  3. Run tests as needed, note that “Run Tests” will not work with the “no default options” checked.
  4. Save and apply settings. 

Resource configuration for distributed computing using a machine file is now complete, and you can proceed to running the simulation.


An example is resource configuration advanced options window shown in the figure below for Windows.

Picture14.png

Using Open MPI: advanced settings (Linux only)

Resource configuration

To configure a distributed computing resource using Open MPI, first follow the instructions above on setting up your cluster and the MPI, then follow these steps:

  1. Create or open your simulation project in the Lumerical CAD/GUI.
  2. Open "Resources".
  3. Change the Name of the resource for reference.
  4. "Edit" the resource.
  5. Select "Remote: Open MPI"
  6. Enter the MPI and FDTD executable in their corresponding fields. 
  7. Check "no default options".  
  8. Enter the total number of MPI processes (-n), hosts (--host), and optionally, the slots per host separate by a colon, e.g. node1:1,node2:2.
  9. Save to apply your changes. 

The resource configuration for distributed computing is now complete, and you can proceed to running the simulation. For more information on command line flags to set processes per node, see Open MPI documentation on Scheduling processes across hosts.

An example of the resource configuration advanced options window is shown below.

Picture15.png

Using a host file

Alternative to using the command line arguments, you can also specify a host file that lists the host addresses and slots for each host and loading the file using the command line argument --hostfile.

An example file is provided below

localhost slots=4
remote2 slots=8
node3 slots=4

To use this file through the CAD window, follow the steps below:

  1. Follow Steps 1 to 7 above. 
  2. Enter the "--hostfile" argument followed by the machine file's filename in the "extra command line options", under "MPI options".
  3. Run tests as needed, note that “Run Tests” will not work with the “no default options” checked.
  4. Save and apply settings. 

Resource configuration for distributed computing using a machine file is now complete, and you can proceed to running the simulation.

An example of the resource configuration advanced options window is shown below.

Picture16.png

Running from the terminal

  1. Open Terminal.
  2. Change the directory to the location of your simulation file.
  3. Set the MPI environment. We are using Intel MPI in Linux in this example.

  4. Run using the MPI runtime executable with the corresponding MPI processes (-n) and ranks per node (-ppn) and remote machines (-hosts) and the corresponding FDTD engine executable. 

    cd /path/to/simulationfile/
/opt/intel/oneapi/mpi/latest/env/vars.sh 
mpirun -n 16 -ppn 4 -hosts localhost,host2,host3,host4 /opt/lumerical/v252/bin/fdtd-engine-impi-lcl -t 1 simulationfile.fsp

Loading a host file or machine file with the terminal

Similar to when running from CAD/GUI, you can also use a machine file (for Intel MPI) or a host file (for Open MPI) when running MPI from the terminal. When using a machine file, enter the corresponding argument followed by the filename and the corresponding FDTD engine executable.

The example below demonstrates this usage for the Intel MPI.

cd /path/to/simulationfile/
/opt/intel/oneapi/mpi/latest/env/vars.sh 
mpirun -machine /path/machinefile.txt /opt/lumerical/v252/bin/fdtd-engine-impi-lcl -t 1 simulationfile.fsp

Calculating how many solver licenses are required

The number of solver/accelerator/HPC licenses required to run your jobs depends on the number of cores used, the number of concurrent simulations, the type of job, and the type of license you have purchased.  See the Understanding solver, accelerator, and HPC license consumption page for details. 

Issues with MPI

Please refer to the OpenMPI documentation or Intel MPI support for issues related to running simulation jobs with MPI. 

 

See also

Compute resource configuration use cases – Ansys Optics 

Resource configuration elements and controls – Ansys Optics  

Lumerical solve, accelerator and Ansys HPC license consumption – Ansys Optics  

Configuring resources for parallel jobs across several computers on Windows – Ansys Optics

Associated files

Related articles