FDTD-GPU Performance Benchmarks

In this article we provide GPU performance benchmarks to demonstrate how they can accelerate FDTD simulation performance by orders of magnitude. This demonstration leverages the same files and some of the benchmark data provided in FDTD Performance Benchmarks – Ansys Optics. To start we will briefly review how to measure computing performance in this context, and discuss choosing appropriate file for benchmarking.  Next the most important hardware factors are highlighted and we contrast CPU and GPU on these metrics. Finally, the technical specifications of a handful of hardware instances are provided before the results are discussed. Throughout we try and provide helpful advice for how minimize runtime and improve throughput.

For more information on how to get started using FDTD on GPU see these links.

• Getting started with running FDTD on GPU – Ansys Optics
• Running GPU simulations – Ansys Optics
• Accessing GPU results – Ansys Optics

Measuring Performance

The most useful data for measuring performances is the ‘simulation wall time’, the real-world time required to run the simulation, and ‘total FDTD solver speed’ which is measured in MegaNodes\second [Mn\s] ie millions of cells computed per second. The simulation wall time, T, is dominated by the simulation run time, T_s, but other processes like meshing and saving the results also contribute the overall time. The simulation time is related to the total solver speed, r, and simulation size, M, by the formula.

$$ T_s = \frac{M}{r}*N  \text{eq. 1}$$ 

The number of iterations, N_iter, is just the total number of time steps. Smaller simulations run faster, as you would expect, and the solve rate depends mostly on the hardware it is running. Large simulations running on older computers can translate in simulations that could take days, but as we will see GPU allows you to reduce this to hours or minutes.

All of this and other important information can be found in solver log file - eg. benchmark_small_p0.log. The log file is where the FDTD program writes information as it executes, and a summary of the benchmark information will be output as the solver terminates.

The log file provides the most verbose simulation information, but performance metrics are also accessible in the FDTD simulation object from the GUI. In this article we share the overall wall time and total FDTD solver speed.

Choosing a Simulation

Simulation time depends mostly on the electrical size of the simulation. Obviously larger simulations require more computation, and it is more accurate to say electrical size because the grid size, and consequentially total number of cells, is a function of wavelength in question. Additionally, FDTD is a time domain method meaning that although the spatial fields can be computed in parallel, the algorithm must proceed sequentially in time. This has a few consequences, first high-Q (or resonant) devices trap light and need to run for longer so that the impulse response can decay. This affects the number of time steps, N in eq 1, resulting longer simulation times. Next, the degree to which you can increase the solving rate by adding more compute resources will tend to saturate. This can be seen by monitoring the solve rate as you add more compute power, say by distributing the workload across multiple computer. Often the solve rate will increase consistently before reaching a plateau, and where this point is depends mostly on the simulation. It is common to see with smaller simulations, but may be effectively out of reach for larger ones. The takeaway is that depending on the size and communication overhead, such as between cards or compute nodes, you may see diminishing returns as you add more compute.

This brings us to another important concept, simulation throughput. Many workflows require analyzing tens or several hundred design variations, in which case maximizing the solve rate for a single simulation may be counterproductive. If your goal is getting through your workload as efficiently as possible, then operating in this region of diminishing returns should be avoided. Instead use concurrency, solving multiple simulations in parallel to better effect. For example, if you have 96 CPU cores, and find that for a given simulation you see the solve rate plateau around 16 cores, then you can run six simulations and achieve nearly 6x speed up.

It is helpful to understand that simulation objects like monitors, sources, and complex materials will all introduce some computational overhead and increase the memory requirements. One of the most powerful aspects of simulation is that you can measure everything you want; however, when maximizing for performance reducing the collected data can help. When possible, you may want to the compare performance using slightly different configurations and settings. See - Getting the Best FDTD Performance – Ansys Optics.

Using a representative file for your workflow that has already been configured appropriately, will usually provide the most useful benchmarks, as this will allow you to compare realistic performance statistics on different hardware and optimize for throughput. Optionally, one may include economic metrics with your benchmark calculations to make the best business decisions.

For our tests, we used 2 simulation files:

1. CMOS image sensor (benchmark_small.fsp): The file requires about 5GB of RAM and runs in about 10-15 minutes on a 32-core desktop CPU
2. Metalens (benchmark_large.fsp): The file requires about 50GB of RAM and runs in about 2h on a 32-core desktop computer.

Hardware Instances

Above we described how to choose a simulation and how certain aspects of the simulation affect performance. It's true that the simulation wall time and solve rate depends on the simulation itself; however, performance is very much hardware dependent. Meaning better hardware will perform better on pretty much any FDTD simulation. There are several things to consider when determining the optimal hardware configuration and this is discussed at length in this article - Lumerical hardware specifications guidelines – Ansys Optics. In this section we will summarize the fundamentals for running FDTD on CPU and then compare that with FDTD on GPU.

On CPU the first thing to understand is that the simulation size is limited by the available RAM. If the simulation cannot fit into the systems RAM the operating system will fall back on auxiliary memory, like the SSD, significantly reducing performance. If we operate within this limitation the next most important factor is the memory bandwidth of the RAM. The FDTD algorithm requires significant data to be passed between RAM and the compute cores. The memory bandwidth depends on the data bus width, transfer rate and number of channels. Newer memory standards, like DDR6 will have higher theoretical memory bandwidth limits, and therefore will produce faster solve rates. Faster clock speeds, and more channels can also improve solve rate by increasing the memory bandwidth. Hyperthreading, has no affect on memory bandwidth and is not recommended. Greater RAM capacity will allow users to run larger simulations, but simply adding ram does not change the machines memory bandwidth or solve rate.

GPU-accelerated FDTD benefits from the greater memory bandwidth available on GPU, as the compute cores and dedicated memory (VRAM) are much more tightly integrated using different memory standards like GDDR and HBM. These architectures typically have memory bandwidths 50-100x greater than CPU system memory; however, VRAM capacity is typically much lower and costs roughly 10x the price of system RAM.  Furthermore, GPU is an additional cost not a replacement for RAM since meshing, data preparation, and post-processing still leverage the CPU. For this reason, GPU simulations must satisfy both system and graphics memory requirements, and the overall simulation time can still be bottlenecked by these CPU processes. As we will show FDTD on GPU provides significant improvements in solve rate and wall time, and is an excellent strategy for accelerating design cycles through high performance computing

For these benchmarks the following compute instances were used.

• C6i - AWS compute-optimized instances, powered by 3rd generation Intel Xeon Scalable processors. This was used with 32 cores and 256 GB RAM. This CPU uses DDR4 RAM with memory bandwidth is estimated to be about 13GB/s.
• C6a  - AWS compute-optimized instances, powered by 3rd generation AMD EPYC 7003 processors with 48 CPU cores. It can use 384GB of RAM with 10GB/s memory bandwidth using DDR4 memory.
• RTX PRO 5000 - A professional laptop GPU based on NVIDIA’s Blackwell architecture. Each GPU has 24 GB of dedicated GDDR7 ECC memory, 82 SMs, and peak memory bandwidth of 896 GB/s. 
• NVIDIA A40 - Ampere-generation data-center visualization and compute GPU. Each GPU has 48 GB of GDDR6 ECC memory, 84 SMs, and peak memory bandwidth of 696 GB/s.
• NVIDIA L40 - Ada Lovelace data-center GPU for visualization, simulation, and compute workloads. Each GPU has 48 GB of GDDR6 ECC memory, 142 SMs, and peak memory bandwidth of 864 GB/s.
• NVIDIA A100 - Ampere data-center compute GPU in the 80 GB PCIe configuration. Each GPU has 80 GB of HBM2e memory, 108 SMs, and peak memory bandwidth of 1,935 GB/s.
• NVIDIA H100 - Hopper data-center GPU in the H100 NVL configuration. Each GPU has 94 GB of HBM3 memory, 132 SMs, and 3.9 TB/s of memory bandwidth.

Benchmark Results

Small Benchmark

For the benchmark_small, the RTX PRO 5000 Laptop GPU outperforms the older A40 by approximately 1.5x. These two GPUs have a similar number of SMs and therefore similar licensing requirements, while the RTX GPU provides approximately 1.3x higher peak memory bandwidth. Both GPUs complete this simulation about 4-5x faster than the tested CPU instances. Memory bandwidth alone does not directly explain the CPU vs GPU performance difference because meshing and other parts of the workflow remain CPU-based. However, it provides a useful reference when comparing GPUs of a similar scale. Overall, the approximate 5x speedup over the compute-optimized CPU instances is significant

A more effective strategy on CPU would be to run several simulations in parallel on these CPU machines. Using concurrency, we could probably achieve a similar solving rate running 4-6 small simulations in parallel. This benchmark shows the maximum solving rate, but CPU allows much greater ability to use concurrency, and for a simulation this size it could be an effective strategy. Finding, the right number of cores to use for this simulation would require some testing but given that RAM available on the c6a(384GB), and c6i(256GB) we could potentially run 12 of these simulations in parallel. This highlights a key difference between GPU and GPU, for the later you need to use the entire GPU card at once, but if you had multiple cards, you can obviously run simulations on each of them in parallel.

Large Benchmark

For this benchmark we see that the RTX 5000(24GB VRAM) suffers a performance penalty. This is because the large simulation(50GB) doesn’t quite fit into the dedicated memory. The card must fall back on the shared GPU memory, and the solve speed will now be rate limited by this secondary memory. The RTX card still achieves solve rates and sim times comparable to the CPU instances, finishing in about an hour, which suggest that, although it may not be advisable, it is a feasible strategy to run simulation on GPU when one lacks the enough vRAM, but has enough shared GPU memory. On CPU we are operating well below the RAM limit, so there is potential for parallelization, but we would likely see the performance drop, so concurrency may not be useful for this simulation. In terms of license utilization, the CPU will use less licenses than GPU typically. The license usage is determined by the number of CPU cores used, or the number of streaming multiprocessors on GPU.

The A40 beats the RTX card, and the CPU instances, by nearly 3x. The A40 has sufficient VRAM, but doesn't quite achieve the same amount speedup in the last benchmark.  Next, we looked at the L40 card which has the same amount and kind of memory as the A40 but uses a newer architecture and has greater memory bandwidth. A speed up of about ~1.2 vs the A40 is observed which is comparable to memory bandwidth ratio. The L40 has a higher number of SMs 142 then any card tested here, meaning it will use more licenses. This something to be aware of if you are license constrained. By utilizing multiple L40s we can increase the solve rate, and we see that when this simulation distributes well across 1-4 L40 cards and see near ideal scaling of the solve rate. The A100 is an enterprise card with HBM, and over twice the memory bandwidth of the L40, as we would expect we see a solving rate greater then x2 the L40. It uses only 108 SMs versus 142 SMs on the L40, so it is better option from a licensing stand point as well. Two A100 on the same machine can then double the solve rate, and we are now looking at simulation time of about 6 minutes. Compared to 1 hour on CPU, we see a 10x speed up with this GPU configuration. 

Finally, the H100 provides similar solver performance as 2xA100, which is expected given the memory bandwidth of 3.9 TB\s ( vs 1.9 TB\s  on the A100). We notice that the total time is actually shorter on the H100, which may indicate that other processes are running faster on this particular machine. If we use two H100, the solving rate doubles again as we would expect; however, the time isn't reduced by the same amount. This highlights that those additional processes, which are not influenced by the solve rate, contribute more with this amount of high performance computing. In comparison with the CPU instances we ultimately see a 16x speed-up with this x2 H100. Also given that the H100 and A100 will require a similar number of licenses to the A100, these H100 cards are probably optimal. All that being said on all GPU cards show significant performance enhancement and any GPU cards that you have available and meets the memory requirements can be used to accelerate computer aided design workflows.