In the automotive market, Light Detection And Ranging sensors (LiDAR) are more and more common. The LiDAR systems present advantages to cover more Advanced Driver Assistance System (ADAS) use cases, where existing sensors such as cameras or radars may present limitations when facing difficult conditions (tough weather conditions, night-time situations).
The DIN standard establishes a measurement methodology for LiDAR sensors, independent of the design of the sensor or the technological approach. This standard is applicable to car manufacturers as well as sensors suppliers, to allow a defined assessment of the sensor performance on a point cloud level. The standard does not seek to establish functional safety requirements. It is intended for R&D personnel, sensor hardware, software developers, automated vehicles manufacturers or testing organizations.
Software prerequisites
To be able to use this example, the following tools and assets need to be installed on your computer:
- Ansys Speos 2024R1
- Ansys Speos Sensor Tools (available with the project's dataset attached)
Overview
Understand the simulation workflow and key results
LiDAR systems use the reflection of transmitted electromagnetic waves, in wavelengths ranging from 850 nm to 1 600 nm, for the measurement of distances (ranging). LiDAR is based on laser technology. These sensors offer high potential due to their precise distance sensing and point cloud image generation. Different technological approaches are leading to a variety of sensors based on the LiDAR principle. Although all these sensors are referred to as ʺLiDARʺ, the differences in the components and technology makes a comparison of the specifications and performance difficult. A standardized LiDAR sensor assessment methodology does not exist up to now.
This document shows the virtualization of the previously defined standard, in a Speos environment. In this example, we focus on the cross-domain test. This test analyzes the robustness of the sensor when the sensor is exposed to multiple targets of differing reflectivities, as well as interfering light sources directly along the sensor axis. The test assesses the change in performance of the sensor under “overload” conditions as compared to baseline conditions.
The evaluation metrics include Key Performance Indicators (KPI). They are described in clustering detections, whether they are positive (on target), or negative.
Run and Results
Instructions for running the model and discussion of key results
The project is prepared in advance - all targets as well as materials are applied already. The LiDAR system used in the example is from the internal Ansys library. It is inspired by datasheets available online, provided by the suppliers.
The step-by-step process described below will follow the creation of the different targets.
As an output, the Speos simulation generates the raw time of flight signal coming back to the sensor. However, as stated in the introduction, the standard measurements are made on the point cloud level. The data generated by the Speos simulation needs to be post-processed. As defined in the previous section, there is an additional tool required, on top of Speos itself which isn't installed with the Ansys Speos installer. Through this add-in, Speos Sensor Tools , Ansys provide a set of default methods, to convert the time-of-flight data into a point cloud. It is important to note that the add-in is not part of the initial product. The Ansys method uses a basic maximum peak detection, without any other post-processing on the data itself.
The standard describes a list of KPIs to measure the performance of the LiDAR.
Step 1: Scene setup, optical characteristics
In the Ansys Speos software, we work in a 3D environment. Most CAD geometry formats are compatible to be imported into an Ansys Speos project. It is also possible to easily generate any geometry, from small components to large infrastructures.
To replicate the standard into Ansys Speos, a simple straight road is created. All targets are also created the same way, according to the standard definition, same orientation, same position every 10 meters.
One strength of a virtual environment is that the ground truth is well known. It means the positions of all objects can be measured and guaranted (no external disturbances such as wind or tolerancing errors in a simulation).
Once all geometries are set up, the first step to prepare a Speos simulation is to apply optical properties on them, which will decide: how the Monte Carlo algorithm will propagate the rays, how much light will be absorbed or reflected, and in what direction.
To achieve a more accurate representation of the material targets, we can get a measurement on these real targets to capture the actual optical properties. For this first draft of the project, we will use assumptions and simplified definitions.
Ansys Speos propose multiple editors to generate the input files, from simplified ones to advanced models. In this project, we use the ‘Simple Scattering Surface’ editor.
In this interface, we define the absorption values, as well as the reflection profile. In the example above, the material is absorbing 10% of the incident light, and the 90% reflected are mainly gaussian, with a smaller part being Lambertian as well as specular. Below is a description of the 3 modes of reflection.
We know from the standard definition that the targets are made of opaque material, therefore no transmission values are being defined for the target materials.
This is a simplified approach since there is no information regarding incident angle and no spectral information; the material will react this way for all wavelengths.
This definition is used 3 times for the targets A, B and C, at 10% Lambertian, 50% Lambertian and 80% Lambertian, respectively.
The most advanced material used in this project is the retro-reflecting one. In the project, we are using an existing Ansys measurement material, which has been measured in the lab on a real road sign sample. The viewer 'BSDF Surface’ allows to visualize the reflection profile. In the picture below, the incident ray arrives at an incident angle of 40°, for an absorption of 78%. And the back-specular reflection towards the same direction is critical, as it will heavily impact the signal coming back to the LiDAR system.
Step 2: LiDAR sensor setup
The second main part of a Speos simulation is the sensor model definition. It will define how accurate and close to the real system measurements the simulation will be. From the Ansys Sensor Tools add-on, we are using a LiDAR reference, inspired from an online datasheet.
The example used in the project is a rotating LiDAR system. There is a mechanical part scanning a certain narrow field of view, that is then repeated across the full horizontal 360° field of view. The accuracy of these two 'firing sequences' files are key to achieve a good simulation.
Below is the scanning pattern, containing 128 points. Both axes are in degrees.
The rotating pattern is a repetition across the 360° horizontal field of view, every 0.1deg.
The firing sequence field can contain a *.txt file, describing each beam's angular propagation, their energy and their time of fire.
Then, the spectrum is defined, through a singular monochromatic value, or a *.spectrum file, defining a slight wavelength divergence of the laser-emitted beams (file editable through text files). This information will be applied to all beams of the firing sequence.
Once the spectrum is done, the next parameter is the intensity distribution of the beams. This parameter is either described by gaussian FWHM (Full Width at Half Maximum) values for both directions, or with an .ies file. This information will be applied to all beams of the firing sequence.
Another more complete version of the firing sequence can contain a dedicated *.OPTScanSequence file, that describes each beam's angular propagation – same as the .txt does – but also a unique wavelength as well as intensity distribution per beam. In summary, each beam can be defined independently from the other beams.
The next parameters are related to the receiver side of the system. First, the optics in front of the receiver are described through a reduced order model taking into account the distortion. This information can be exported from an Ansys Zemax OpticStudio project, to be as close as possible to the real system characteristics, as well as protecting the IP of the optical supplier. It can also be created from a datasheet, with the right information provided.
The full transmittance of the optics is a percentage factor; no further model is available at the time writing this article (Speos 2024R1).
Focal length and aperture pupil size are dimensions defining the optical system.
The imager is defined by its size (Width and Height) and Resolution. The Resolution is still a Software Beta feature, which can be activated in File/Speos Options/Advanced/General, then tick the box ‘Enable beta features’. In case the resolution isn’t active, the sensor is considered as a single pixel receiver. It greatly increases the efficiency of the simulation, by reducing the size of the output file.
Spatial accuracy is a key parameter of the simulation. It describes the sampling of the signal (red bars on the graph below) on the time domain axis. Since the simulation is time-of-flight based, the returning signal (green signal on the graph below) is power relative to time. The link to spatial accuracy is the conversion with light velocity.
The Start and End parameters define an ‘area of interest’. All geometries outside of this zone will not be picked up by the LiDAR simulation. It is a great tool to focus the analysis on a specific area, without considering the whole scene, again to increase the efficiency of the simulation (size of the output file).
The last option in the LiDAR model definition is the possibility to add a geometrical aiming area. It is a simulation tool dedicated to gathering the rays coming back to the receiver system more efficiently. It is mainly used when there is a cover in front of the receiver. For this project, it will not be active.
Step 3: Run simulations.
Once the geometries are set, and the LiDAR model defined, it is time to manage the simulations. The LiDAR simulations require the package Speos Premium Sensor, or Speos Enterprise.
Depending on which test method is expected, we select the required target(s) from the CAD Structure Tree. In the example, only one target is selected. Intentionally, the road is not added in the geometry selection, to prevent the output point cloud from being polluted by many points on the road. It is assumed that the road does not impact the signal returning from the targets. We apply the same approach for the road markings. Then we select the LiDAR model previously defined in the sensor selection.
The default 'General' parameters are fine to keep them as is. One parameter that is key to understand well is the 'Number of rays.' This number equates to the number of times the full firing sequence will be run.
Example, if the firing sequence contains 1000 points covering the LiDAR system field of view, and the factor in the general parameters is set at 1000 rays, it means that the raytracing will calculate 1000 x 1000 rays, so 1.000.000 rays to be computed. Therefore, a factor of 1 will be enough to run a preview of the LiDAR system emitting firing sequence. It has a direct impact on the duration of the simulation.
In the example presented in this article, two simulations will be consecutively run with only the factor at 1. This way, it will highlight the difference between two ‘frames’ of the real LiDAR system.
In the results selection, there are 3 options. The Field of view and the Map of depth are only available for the Flashing type of LiDAR, so the only output that we keep is the Raw time of flight.
Once the simulation is completed, there is the expected raw time-of-flight file, plus an .html document to summarize the simulation report (geometries embedded, simulation error percentage, etc).
Step 4: Post-Process LiDAR outputs results.
As described in the introduction, the post-process methods are not natively part of the Speos software. The add-on "Ansys Sensor Tools" covers that need. If the user has access to scripting capabilities, and knowledge on the algorithm that is embedded on the LiDAR system, then the raw information from the time-of-flight is accessible through APIs, available on the Speos documentation pages.
In both cases, the objective of the post-process is to generate a point cloud, on which the measurements defined in the SAE Standard will be applied.
Sensor Tools Add-on
Available from the Ansys Store website, a new ribbon is added and brings in a new set of features dedicated to sensors simulations – both for camera and LiDAR.
Within this project, the main tool is Sensor Tools/Signal Processing. It takes the raw time-of-flight data as input, and can generate the point cloud, as well as a .csv file that can be read in an editor. The signal processing is applying a simple algorithm to detect the max peak. Other post-process options exist to add other effects, such as noise or ambient lighting contributions.
Once the point cloud file .pcd is generated, a dedicated viewer will allow the user to visualize the point cloud file. Few parameters exist to define the point cloud visualization options (intensity or distance information, scale color mainly).
It can also be imported into the 3D scene. To do so, another feature from the Sensor tool is required, ‘PCD import’. It is possible to import multiple iterations of point cloud data, which is what is used in this example: import of 2 point-clouds data, and the measure of the difference between the points positions on the target or its vicinity.
On the example above, we can highlight the difference and gaps between 2 point-clouds on the 3D view directly, as well as a margin of error regarding the boundary box of the target.
Step 5: DIN Standard measurements.
The standard describes different types of metrics,
- Detection distance
- Resolution
- False-positive detection
- Spatial error
- Separability
In the simulation example provided together with the dataset, we will focus on the spatial error metrics, particularly the KPI 9, the angular precision. It reads as follows:
“Every measurement frame is processed by averaging the centers of all true-positive clusters, featuring at least one detection in this frame. The angular precision (AP) is the result of standard deviation calculation using this series of averaged centers in azimuth and elevation.”
The graph below represents the data for 4 frames, or rays. We can observe points where gaps appear. The main cause of this gap is the laser divergence of the LiDAR.
The two axis on the graph below are representating the angular position in degrees of the point cloud data on the target. Therefore we read the sensor is in the same horizontal plan as the center of the target (Y axis centered on 0) and slightly on off-axis (X axis centered on 5°)
To calculate the KPI 9, we focus on this area in particular:
The standard deviation for these 4 points is then:
Important Model Settings
Description of important objects and settings used in this model
As described in the step-by-step workflow above, the key parameters for a LiDAR study are:
- Sensor settings: laser light source definition, optical lens parameters, position, …
- 3D Scene setup
- Simulation settings
- LiDAR Post-processing settings
Updating the Model With Your Parameters
Instructions for updating the model based on your device parameters
More methods and metrics will be added in a later updated version of the article, once the required equations have been discussed and reviewed with the people in charge of the standard document.
Others KPI defined in the standard are compatible with Speos simulations as well, the attached dataset is mainly a sandbox project, where the user can perform its own performances tests.
Additional Resources
Additional documentation, examples and training material
Relevant Speos Ansys Learning Hub courses:
-
- Ansys Speos Getting Started
- Ansys Speos Optical Sensor Test