Using GPU for simulations in Ansys Lumerical FDTD™ can significantly speed up your simulations.
This page discusses how to set up resources for GPU simulations on a single node (local or remote) and important settings for single-node multi-GPU, as well as threading. For multi-node multi-GPU simulations, additional setup is required, and is shown in the Resource configuration for multi-node multi-GPU simulations Knowledge Base article.
Setting up a GPU resource
To set up a GPU resource, follow the steps below:
- Open the Resource Configuration window from the File tab of Lumerical FDTD
- Ensure a GPU device is listed and activated in the resource list; on new installations, a new GPU resource will be created if at least one GPU is present. Click Add to add an additional device.
- Select the Device Type for your GPU device:
- GPU: Automatically selects a GPU to use.
- GPU Custom: Allows you to select specific GPU(s) to run the simulation in the advanced settings. Further details are discussed in the subsection below.
- GPU X: Selects a specific GPU on the system to use. This only applies to the local computer. For remote resource, use the CUDA_VISIBLE_DEVICE environment variable.
- Customize the name of the GPU device if required, to help distinguish between different resources.
GPU resource configuration is now complete, and you can further customize the resource as needed or proceed to running the GPU simulation.
Specifying the GPU device to use
When you have multiple GPU devices in the system, the solver uses all GPUs by default. You can specify the device to use with the CUDA_VISIBLE_DEVICES environment variable. The resource configuration advanced window also offers options to adjust it.
To set the CUDA_VISIBLE_DEVICES variable on a local machine with the resource configuration advanced window, follow the steps below:
- Change the Device Type of the resource to GPU Custom in the Resource configuration window.
- Click Edit to open the Resource advanced options window.
- Select your desired GPU devices using the gpu devices field in FDTD options in the opened window.
The CUDA_VISIBLE_DEVICES setup is now complete, and you can proceed onto running the simulation on your local machine.
For remote nodes, you can select the specific GPU(s) to use by explicitly setting the CUDA_VISIBLE_DEVICES environment variable. For example, when using the Microsoft MPI, you can provide an extra command line option, /env CUDA_VISIBLE_DEVICES 3 to select GPU 3 on a remote machine.
Threading and CPU when using the GPU
When you use GPU for FDTD simulations, operations other than the solver still use the CPU. Examples of these operations include script commands, and farfield projections.
To set the number of threads for operations during the solve process, you can use the Threads column in the Resource configuration window according to the number of cores on the resource. This should be set to the minimum value between the number of cores allowed by your licenses and the number of logical cores on the CPU. Setting this column to auto allows the software to use as many threads as number of licenses checked out, which is determined by the number of streaming microprocessors in the GPU.
To set the number of threads used for operations that occur in the design environment, including scripting and farfield projects, you can use the “Design environment” tab in the resource configuration window.
For more information on setting CPU threads, see the Knowledge Base article on Resource configuration elements and controls.
Script access
You can also set up the resources using the script command setresource.
setresource("FDTD", 1, "device type", "CPU");
setresource("FDTD", 2, "device type", "GPU"); #< any dropdown option can work in place of “CPU” or “GPU”, such as “GPU custom”, “GPU 0”…etc.
Note: For remote nodes where you want to control the specific GPU used, use the “GPU Custom” Device Type, and directly set CUDA_VISIBLE_DEVICES variable, either through command line options, or using the gpu devices field in the advanced options.
See Also
Getting started with running FDTD on GPU, Resource configuration for multi-node multi-GPU simulations, Running GPU simulations