Running simulations with MPI on Linux

As of 2024 R1, MPICH2 is no longer packaged with the Lumerical installation package on Linux. For best performance and distributed solve capability, it is recommended to install either Open MPI or Intel MPI.

This document outlines how to install these packages and configure Ansys Lumerical to use them.

Important

Root or sudo access is required for installing packages from your system package manager.
For distributed computations (multiple nodes), you will need to either install these packages on all your nodes or make sure they are in a shared file system mount.
Installation instructions are for the latest released versions of the Linux distributions used in the examples.

Intel MPI installation

The following instructions will walk you through,

Installing required packages,
Adding the Intel MPI to your package manager,
Installing Intel MPI, (example shows version 2021.16 of Intel MPI).
Configuring your environment to run Lumerical.

RHEL/Rocky/Amazon Linux (dnf package manager)

sudo dnf -y install dnf-utils
sudo dnf config-manager --add-repo https://yum.repos.intel.com/oneapi
sudo rpm --import https://yum.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2023.PUB 
sudo dnf -y install intel-oneapi-mpi-2021.16

Ubuntu (apt package manager)

sudo wget -O- https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2023.PUB | gpg --dearmor | sudo tee /usr/share/keyrings/intel-oneapi-archive-keyring.gpg > /dev/null
sudo echo "deb [signed-by=/usr/share/keyrings/intel-oneapi-archive-keyring.gpg] https://apt.repos.intel.com/oneapi all main" | sudo tee /etc/apt/sources.list.d/oneAPI.list
sudo apt-get update
sudo apt-get install -y intel-oneapi-mpi-2021.16

SUSE Linux Enterprise Server (zypper package manager)

sudo zypper addrepo https://yum.repos.intel.com/oneapi oneAPI
sudo rpm --import https://yum.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2023.PUB

sudo zypper install intel-oneapi-mpi-2021.16

Intel MPI environment configuration

You must first set up the environment for Intel MPI before you can run Lumerical simulations with it. The easiest way to set up the environment is to first run a script that is provided with the installation and export the UCX_TLS variable.

## example shown for Lumerical stand-alone installer
export PATH=$PATH:/opt/lumerical/[[verpath]]/bin
source /opt/intel/oneapi/mpi/2021.16/env/vars.sh
export UCX_TLS=ud,sm,self

You can set this permanently by "sourcing" these into your ".bashrc",

## example shown for Lumerical stand-alone installer
echo “export PATH=$PATH:/opt/lumerical/[[verpath]]/bin” >> ~/.bashrc
echo " source /opt/intel/oneapi/mpi/latest/env/vars.sh" >> ~/.bashrc 
echo "export UCX_TLS=ud,sm,self" >> ~/.bashrc

In addition to running the script above, you must also copy the library files into the Lumerical installation “lib” folder. A sample command is shown below, change the path according to the Lumerical installation on your machine.

## example shown for Lumerical stand-alone installer
sudo cp /opt/intel/oneapi/mpi/2021.16/opt/mpi/libfabric/lib/libfabric.so.1 /opt/lumerical/[[verpath]]/lib/
sudo cp /opt/intel/oneapi/mpi/2021.16/lib/libmpi.so.12 /opt/lumerical/[[verpath]]/lib/

Open MPI installation

Note: For distributed computing using GPU and Open MPI, please refer to the section below on CUDA-aware Open MPI.

The following instructions will walk you through,

Installing Open MPI. (example shown install OpenMPI v4)
Configuring your environment to run Lumerical.

RHEL/Rocky/Amazon Linux (yum package manager):

On RHEL 8 and and up, and on Rocky 8, use the package name openmpi, which will install version 4

sudo yum install openmpi

Rocky 9 (Manual install)

On Rocky 9, install Open MPI manually using the following instructions:

Download the tarball source from its official website.

wget https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.1.tar.gz

Install the gcc compiler if it has not yet been installed.
```
sudo yum groupinstall "Development tools"
```
Uncompress the tarball.
```
tar -xzvf openmpi-4.1.1.tar.gz
```
Navigate to the OpenMPI folder and set up compilation for install
```
cd openmpi-4.1.1
./configure --prefix=/usr/local/
```
If you want to install other versions, define the –prefix argument to that specific directory instead. For example,
```
./configure --prefix=/usr/local/openmpi-4.1.1/
```
Install using the make command.
```
sudo make all install
```

The installation for Open MPI on Rocky 9 is now complete.

Ubuntu (apt package manager):

sudo apt-get install openmpi-bin libopenmpi-dev

SUSE Linux Enterprise Server (zypper package manager)

sudo zypper install openmpi4

Open MPI environment configuration

When using Open MPI, the solver must be able to find the MPI runtime libraries, and the Lumerical product executables must also be visible to the MPI executable.

RHEL/Rocky/Amazon Linux

Assuming a default install location of "/usr/lib64/openmpi/lib", one way of allowing the solver to find the MPI runtime libraries is to add that installation path to your LD_LIBRARY_PATH environment variable,

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64/openmpi/lib

Adding the "/usr/lib64/openmpi/bin" location to your PATH environment variable will let you call "mpiexec" without specifying the entire path.

export PATH=/usr/lib64/openmpi/bin:$PATH

The above commands could be added to your bashrc to avoid running them again. Detailed documentation can be found on the Open MPI website, https://www.open-mpi.org/doc/.

echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64/openmpi/lib" >> ~/.bashrc
echo "export PATH=/usr/lib64/openmpi/bin:$PATH" >> ~/.bashrc

In addition to configuring for the executable and runtime libraries for Open MPI, the Lumerical product executables must also be added to the PATH environment variable. One method to do it is to append the following to your .bashrc file:

export PATH=$PATH:/opt/lumerical/[[verpath]]/bin

Then, either restart the machine or run the following command

source ~/.bashrc

Ubuntu

On Ubuntu, you need to modify the mpi library path and environment variable options in the resource configurations advanced options to ensure that the program can find the executable and runtime libraries.

/usr/bin/mpiexec
or 
/usr/bin/mpirun

SUSE Linux Enterprise Server

On SUSE Linux Enterprise Server, follow these commands to allow the solver to find the MPI library and executables.

sudo ln -s /usr/lib64/mpi/gcc/openmpi4/bin/mpirun /usr/lib64/mpi/gcc/openmpi4/bin/mpiexec
source /usr/lib64/mpi/gcc/openmpi4/bin/mpivars.sh

Open MPI – multi-node multi-GPU and CUDA-aware Open MPI

Using a CUDA-aware Open MPI set up will maximize the performance of multi-node mult-GPU simulations. We recommend installing the HPC-X package from Nvidia for this use.

When using a CUDA-aware Open MPI, additional setup is required.

For further information on GPU computing with Lumerical FDTD, including how to configure the resource manager for multi-node multi-GPU distributed computing, see this Knowledge Base article.

Matching Your MPI to a Solver

It is necessary to use the version of the solver that is matched to the version of MPI being used to run the solver. The following table lists the engine executables to run with supported MPI.

Intel MPI

fdtd-engine-impi-lcl
varfdtd-engine-impi-lcl
eme-engine-impi-lcl

OpenMPI

fdtd-engine-ompi-lcl
varfdtd-engine-ompi-lcl
eme-engine-ompi-lcl