Parking assistance camera integration

This article showcases a straightforward camera integration workflow for the transportation industry, in this case. Using this simple methodology, one can easily check whether the parking assistance camera meets any regulation standard such as FMVSS 111. This allows the user to take critical decisions for the system design or the system positioning at a very early stage of the conception, saving both time and prototyping costs.

download example

Overview

Understand the simulation workflow and key results

Camera sensors are widely employed in the automotive industry for various purposes, from autonomous vehicle applications such as road and environment analysis, as well as driving assistance applications like driver monitoring systems and park assist cameras. In the latter case, it is of the highest importance for integrators to evaluate camera performance under different vehicle conditions using metrics such as visibility and total field of view. In this article, we will present a straightforward workflow that integrators can use to proactively address potential issues during the camera integration process and have better guidance towards the specifications of the optics and the sensor of the camera.

We will use Ansys Zemax OpticStudio to design and export the optical system, then Ansys Speos will be used to position the camera and check parking regulation standards such as FMVSS 111. Finally, we will use Speos Sensor System Exporter to generate the final sensor image.

Step 1: Lens system design export with Ansys Zemax OpticStudio

In this section, you will learn how to transfer a Reduced Order Model from Ansys Zemax OpticStudio to Ansys Speos.

Step 2: Lens system integration with Ansys Speos

In this section, you will assess the camera performance on the vehicle.

Step 3: Sensor image generation using Speos Sensor System Exporter

In this section, you learn how to generate a camera image or movie file from a simulation result.

Run and Results

Instructions for running the model and discussion of key results

Step 1: Lens system design export with Ansys Zemax OpticStudio

Ansys Zemax OpticStudio is used for the design of the optics of the camera.

1. Open Ansys Zemax OpticStudio.
2. Open “Wide angle lens.zmx”.

The lens design is a wide-angle optical system, since an extended field of view is necessary for parking assistance scenario. The total field of view is 200 degrees.

3. Go to “System Explorer” > “Units” and make sure the lens units are set to Millimeters.
4. Go to “File” > “Export Reduced Order Model to Speos.”

The "Reduced Order Model" tool opens. This tool will write a Reduced Order Model (ROM) of the current system in a “.OPTDistortion” file, a format readable by Ansys Speos for simulation. The file format contains information about the ray direction, the starting point, the emissivity and the focus distance, for every sample point on the sensor.

5. Specify the size of the sensor. Since we will be using the Sony IMX226 in this example, the size will be 7.5332mm (width) x 5.6351mm (height).

At this stage, the preview in the tool gives the indication that the optical system is not calibrated for the sensor size since the sensor does not capture the whole image plane. Hence, the sensor might not be able to capture the total 200° field of view.

6. To get better accuracy in the ROM, increase the diagonal sampling to 40 samples.

Since the behavior of the system is interpolated between sample values, the more sample points we add, the better the accuracy of the model.

7. Launch the calculation.
8. After calculation is done, save the result as “System1.OPTDistortion” file in the “Speos input files” folder.

The “System1.OPTDistortion file” is the ROM of the previous optical system. We will now use this file in Ansys Speos to compute the luminous exposure map reaching the optical sensor.

The workflow benefits from this ROM approach in two ways. First, using a ROM instead of the complete optical system will lead to much faster results while the accuracy remains high if your ROM was setup correctly. Also, it’s worth highlighting that since the lenses geometries and properties are not stored in the ROM, the Intellectual Property of the lens system supplier stays protected. This is perfect for an integration purpose, because usually the integrator will not design the optical system on its own.

The optical effects considered in the ROM are geometrical aberrations (distortion, field curvature, comma, etc.), chromatic aberrations, the field of view (FOV), the depth of field (DOF), etc...

Step 2: Lens system integration with Ansys Speos

Now we will use Ansys Speos to integrate the camera onto a vehicle in a parking environment.

1. Open Ansys Speos
2. Open “Parking camera integration.scdocx”.

The project contains a parking scene matching the FMVSS111 regulation standard. To conform to the regulation tests, the car is parked at a precise distance from the scene geometries to assess the FOV of the park assist camera.

3. Compute the 2 lightboxes of the project to display the elements of the scene.

In this project, both the parking scene and the SUV are integrated into Speos LightBox to save initialization and meshing time in the simulation. For more information on Speos LightBox, please refer to this section of Ansys Speos user guide.

There are also three sensors which represent the same camera. The geometric camera “Camera.Geometric” will be used in an interactive simulation to project the field of view of the camera on the scene while the photometric camera “Camera.Photometric” will be used in an inverse simulation the generate the exposure map from the camera. For more information about Speos camera types, please refer to this section of Ansys Speos user guide.

4. Double click on “Camera.Photometric” to edit the camera sensor.
5. In the Distortion file area, browse for the previously exported “System1.OPTDistortion” file.
6. Double click on “Camera.Geometric” to edit the camera sensor.
7. In the Distortion file area, browse for “System1.OPTDistortion” file.

Now the photometric and the geometric camera are equipped with the ROM from the Ansys Zemax OpticStudio design.

We will now assess whether the camera located on the rear side of the vehicle successfully passes the FMVSS 111 standard. For field of view check, the standard specifies the following:

“When tested in accordance with the procedures in S14.1, the Rearview image shall display:

• • A. a minimum of a 150-mm wide portion along the circumference of each test object located at positions F and G as shown in figures 13 and 14.
    Note: A visible 150-mm wide portion means that the full width of any horizontal segment of the 150-mm vertical stripe must be visible. The entire vertical stripe need not be visible.

• • B. the full width and height of each test object located at positions A through E as shown in figures 13 and 14.”

The real test is checking if the user can see the obstacles from the dashboard’s display. The 5 obstacles located at the back of the scene must be entirely visible, whereas the two front obstacles must be partially visible (we should be able to at least see one vertical portion containing the full width). For more information you can refer to the laboratory test procedure for FMVSS 111 .

The dashboard screen display will not be simulated in this article, but a prerequisite to fulfilling the previous regulation is first that the camera can see the obstacles as specified.

There are 2 ways we can check this in Ansys Speos. The quickest way is by projecting the camera's field of view over the scene. This is done by using a geometric camera in an interactive simulation.

8. Run the simulation “Grid projection”.

After it has run, we can see the camera FOV directly in the 3D scene.

We can see that with this design, we can entirely detect the 5 rear obstacles, but the two front obstacles are not detected at all, which is mandatory to pass the regulation.

We can double check this result by using a photometric camera in an inverse simulation.

9. Run the simulation "Camera simulation".
10. After computation, double click on the “Camera simulation.Camera.Photometric.hdr” result to open it

We can see that we still fail to see the two front obstacles of the scene, therefore it’s unlikely that the driver will see them on the display from the dashboard. This is most likely due to the sizing issue we noticed from Ansys Zemax OpticStudio when we generated the ROM. We can then conclude that with this optical system and sensor, the FMVSS111 field of view regulation is not satisfied.

Since we don’t want to change the sensor used in the camera, we need to scale the optical system to fit the sensor size. By doing so, the sensor will be able to capture more field of view, hence the standard is more likely to be fulfilled.

11. From Ansys Zemax OpticStudio, open “Wide angle lens_scaled.zmx”
12. Go to “File” > “Export Reduced Order Model to Speos.”
13. Specify the size of the sensor: 7.5332mm (width) x 5.6351mm (height).

This time we can see that the total field of view of the system fits perfectly the sensor diagonal size.

14. Increase the diagonal sampling to 40 samples.
15. Save the results as “System2.OPTDistortion” file in the “Speos input files” folder.

Now, let’s load this new model into our Ansys Speos scene.

15. From Ansys Speos, double click on “Camera.Photometric” to edit the camera sensor.
16. In the Distortion file area, browse for the previously exported “System2.OPTDistortion” file.
17. Double click on “Camera.Geometric” to edit the camera sensor.
18. In the Distortion file area, browse for “System2.OPTDistortion”.

19. Run the simulation “Grid projection”.

With the new FOV projection in the scene, we can see that the first two obstacles are now well detected, compared to the previous design.

Let’s confirm this by running the inverse camera simulation.

20. Run the simulation “Camera simulation”.
21. After computation, double click on the “Camera simulation.Camera.Photometric.hdr” result to open it.

The result concurs with the FOV projection. We can now clearly see the two front obstacles as well as the 5 rear ones. With this optical system, the FMVSS111 standard field of view requirement is satisfied.

Step 3: Sensor image generation using Speos Sensor System Exporter

We can use the last simulation results to generate the actual image produced by the camera, which is most likely going to be the one displayed on the dashboard. For this, we can use the Speos Sensor System Exporter (SSS Exporter), a post-processing tool that converts irradiance and exposure maps from Ansys Speos to raw maps and processed RGB images using a reduced order model of the optical sensor. For more information on how to use this tool, please refer to this Knowledge Base article .

In this example, we are employing the Sony IMX226 RGB CMOS sensor.

1. In the folder “SSS Exporter/Inputs”, open the “IMX226_Sensor.yaml” file and inspect its contents. If preferable, modify the sensor operating conditions (lines 7-19), such as the gain and offset value, exposure time, and sensor temperature, into the desired values.

This file contains the EMVA-1288 standard data for linear characterization of the sensor and it has been created based on technical information on the IMX226 retrieved from Sony's technical documentation .

2. From the “Speos output files/Parking Camera Integration” folder, copy the file named “Camera simulation.Camera.Photometric.Irradiance.xmp” and paste it into the “SSS Exporter/Inputs” folder.

This is the irradiance map generated by Ansys Speos using the latest optical system with enhanced field of view. The SSS Exporter will take this irradiance map and process it, simulating how the actual camera sensor would convert this information into electrons, factoring in all associated noise that could potentially impact image quality. This stage can also produce an electron map and a noise map, which are critical for understanding the sensor's performance under various lighting conditions.

3. In the folder “SSS Exporter”, open the “Inputs.yaml” file, uncomment the Raw export and Electron export lines by deleting the hash (#) symbol, and save the file, to allow the SSS Exporter to generate the respective maps.
4. In the folder “SSS Exporter”, double click on “Launch Speos Sensor System Exporter.bat” to launch the post processing.

After the processing is done, you can find the output of the SSS Exporter in the “SSS Exporter/Outputs” folder. Foremostly, this is a raw image in XMP format. This image contains all the data captured by the sensor before any processing is applied. And this raw data is invaluable as it gives us the clearest insight into the pure performance of the sensor.

This raw image is also developed into a final format, which can be used for different applications depending on the requirements. This developed image is what would be presented to the end-user; a post-processed image that has been optimized for visibility, color accuracy, and overall aesthetic appeal.

It's important to highlight that while the final image is what the user sees, the value of intermediate results such as the noise or electron maps cannot be overstated. They provide us with a sandbox to fine-tune our sensor and camera setup to ensure that our final output is not just good, but optimal.

The SSS Exporter also offers the possibility to generate a movie animation consisting of several consecutive image frames as the car moves into the parking position. To generate these frames, we must revisit the Speos scene and run a simulation for each specific car position respective to each movie frame.

Instead of manually adjusting the scene and executing simulations tens of times, it is possible to automate this process using the scripting capabilities of Ansys Speos. The script that performs the necessary operations is already written and available in the “Speos Input files” folder because the automation aspect is outside the scope of this article. However, for more information on the scripting capabilities of Ansys Speos please refer to this section of Ansys Speos user guide.

5. Go back to the Ansys Speos scene
6. Click on File > New > Script
7. On the Script Editor that just appeared, click on the “Open Script” folder-icon and then select the “SimulationUpdate.scscript” file in the “Speos input files” folder.

This script runs a while-loop, each repetition of which updates the start time of the "Movie simulation” before executing it. The parking motion itself is determined by the “Parking_trajectory.json” trajectory file defining timely-wise the Axis System of the “Camera.Movie” sensor that is the one selected by the executed simulation.

8. On the Script Editor, click on the chain-icon to publish the script as Group, define the name of your choice, and click OK.
9. Go to the “Groups” tab and locate the Group you just created. Right-click on it and click “Run Script”.

The script will consecutively execute 36 simulations worthy of 1 second of virtual time, since the frame rate of the sensor is 35 FPS. This operation might take several hours depending on the computer’s capabilities, and each of the resulting exposure maps will have a size of about 750 MB, so about 27 GB will be generated in total.

Note: The available script executes GPU-based simulations. Check your Speos license to ensure use of GPU.

10. Transfer the generated exposure maps from the “Speos output files” folder into the “SSS Exporter/Inputs_animation” folder.
11. In the folder “SSS Exporter”, right click on “Launch Speos Sensor System Exporter.bat” to edit the file with a text editor of your choice (e.g. Notepad).
12. Replace the “Inputs.yaml” argument in the first line with “Inputs_animation.yaml”.

This will instruct the SSS Exporter to process the input files dictated by the latter file to generate the animation frames and, consecutively, the animation itself.

13. Save the “Launch Speos Sensor System Exporter.bat” file, close the text editor, and run it.

The processing will require several minutes. When it is done, you can find the output of the SSS Exporter (36 developed frames and the movie animation) in the “SSS Exporter/Outputs_animation” folder.

The video is shot with a gain of 3 and an integration time of 60 milliseconds for each frame, played at a standard rate of 35 frames per second. This is a real-time representation of what the driver would see through the park assist camera. The image is clear, and the motion is smooth, indicating that the system is well-suited for real-time applications. This animated scene not only validates our sensor settings but also provides us with an experience of the parking assistance system in action. It's the integration of design, simulation, and real-world application that results in a refined product, fully prepared for deployment in the automotive sector.

Important Model Settings

Description of important objects and settings used in this model

• Camera settings:
  • Camera position

Camera sensors, like every other sensor in Ansys Speos, must be in space to record data from a specific project region. A system coordinate is used for positioning the camera, and it references the camera position according to the optical sensor plane. Therefore, you can easily position and move your camera(s) to any desired location of the project. More information about Speos camera sensors can be found here .

• • OPTDistortion file from Zemax

The methodology described to export a reduced order model from Ansys Zemax OpticStudio to Ansys Speos for simulation is applicable to any sequential optical system. This is a good approach to simulating efficiently a camera system for field of view analysis. Indeed, the ROM considers the effect of distortion, depth of field, the entrance pupil position as well as geometric aberrations such as coma or field curvature. However, due to directly propagating the rays from a virtual emissive surface, this approach should be avoided for straylight analysis because inter-reflection and multiple ray paths are not considered. More information about the ROM can be found here .

Updating the Model With Your Parameters

Instructions for updating the model based on your device parameters

• Post processing Settings
  • Variation of sensor files

The SSS Exporter requires some input files to convert properly an irradiance map from Speos. The most important one being the text-based “.yaml” file describing the sensor model. This file contains useful information for linear characterization of the sensor’s response. Intrinsic parameters such as quantum efficiency of the pixels, Bayer Matrix configuration, dark current and dark noise are considered to generate an accurate raw map based on EMVA-1288 characterization standard. Post-processing parameters can also be specified to generate an RGB image. More information about the SSS Exporter YAML files can be found here .

• • Variation of operating conditions

Besides the sensor intrinsic parameters, the “.yaml” file also contains parameters relating to the operating conditions of the sensor (exposure time, temperature, gain, etc.) One can vary these parameters using the same sensor to simulate different acquisition scenarios. More information about this aspect can be found using the link above.

Additional Resources

Additional documentation, examples and training material

• Ansys Speos Getting Started
• Ansys Speos Workflow and Geometry Management
• Ansys Speos Camera Sensor Visualization
• Getting Started with Speos Sensor System Exporter