CMOS Sensor Camera - Image Quality Analysis in a 3D Scene

In this example, we introduce a simulation workflow to analyze the image quality of a camera system from a combination of an optical system and a CMOS imager within a specific environment with different lighting conditions. This example covers mainly the Ansys Speos part of the overall workflow. The optical system is designed with Ansys Zemax OpticStudio and exported to Ansys Speos for system-level analysis. The CMOS imager is designed with Ansys Lumerical and exported to Ansys Speos.

download example

Overview

Understand the simulation workflow and key results

In camera systems, CMOS (complementary metal oxide semiconductor) imager is an electronic component where incident absorbed photons generate a photocurrent that can be digitally processed. In this example, we used Ansys complete optical solution and imported the optical system information from Zemax OpticStudio and CMOS imager from Lumerical into Speos to do a full camera system analysis in a 3D scene and simulate the electronic map, and final image generated by the imager. This interoperability workflow takes into consideration the interplay between the macroscopic camera lens and the microstructure of the CMOS image sensor when simulating the entire optical system. With the Speos capabilities of handling realistic illumination and photometric/radiometric physics-based rendering, users can easily optimize the components and construct an accurate view of the final electronic map and final image as recorded by the image sensor to design application-based cameras.

There are four main tools needed for this virtual solution :

1. Zemax OpticStudio to export lens model designed in Zemax OpticStudio to be used by Speos
2. Speos to generate spectral exposure map in front of the CMOS imager
3. Lumerical FDTD (Finite Difference Time Domain) and CHARGE to compute the quantum efficiency of the sensor as a function of incidence angle and wavelength
4. Speos Sensor System Exporter to combine Speos and Lumerical results and generate the electronic map captured by the sensor as well as the final image

As this example requires Lumerical, Zemax OpticStudio and Speos simulation, we mainly limited ourselves to Speos simulations here as described by the following steps:

Step 1: Lens system design with Zemax OpticStudio (not covered in this article)

In this step, we design the camera lens system in Zemax OpticStudio and export the reduced order model of the lens system using the "Export to Speos Lens System" feature from OpticStudio (*.OPTdistortion file). The reduced order model (ROM) generated lens model is called Speos Lens System (SLS). This operation is done once for the lens system which enables us to perform fast ray tracing camera simulations in Speos afterward.

Here we use a compact and efficient lens system for mobile phone cameras designed with Zemax OpticStudio. The incident angle at the sensor surface increases to 35 degrees at about 2 mm distance from the center of the sensor and then decreases to 25 degrees at the corner of the sensor.

SLS can be generated by Zemax OpticStudio automatically for symmetric lens system or can be created from Chief ray angle curve information from a datasheet information. (if the lens design information is not available).

Step 2: Speos Simulation

The *.OPTdistortion file is imported into the Speos camera sensor to define the lens performance of the camera system and evaluate sensor perception in a 3D scene with realistic illumination conditions. We run a ray tracing photometric ROM camera simulation which is about 100x faster than the full lens system simulation in Speos and extract the key imaging metrics such as spectral irradiance and exposure map. The spectral irradiance map provides a detailed visualization of the amount of light that reaches each point on the sensor in W/m²/nm, which is essential for determining the quality of an image captured by a camera. It can help identify hotspots, shadows, and other lighting anomalies that can affect the overall image quality. In addition to that spectral exposure map gives information about the amount of light in Joules/m²/nm that reaches the camera sensor. It helps to determine the optimal exposure time settings for the camera to capture a high-quality image. In this example the spectral exposure map in front of the CMOS imager sensor is calculated from the full 3D scene shown below under different ambient lighting conditions including day, night, and night with a lamp.

Step 3: Lumerical Simulation - FDTD, CHARGE- (not covered in this article)

The spectral exposure map simulated by Speos in front of the CMOS imager needs to be combined with the sensor’s quantum efficiency to generate the electron map. Lumerical FDTD and CHARGE tools have been used to quantify the quantum efficiency of the designed CMOS sensor. The CMOS image sensor is composed of microscopic pixels with optical and electronic elements. The main optical elements are micro-lenses and color filters used to focus light of the desired wavelength on the correct spot in the silicon substrate at the bottom of the imager. The absorbed photons generate charged carriers that are collected and transferred for detection on the electronic side. The electronics have components including gates and interconnects which can interfere with the path of light inside the sensor. The coupled optoelectronic simulation is done in FDTD and CHARGE.

Step 4: Speos Sensor System Exporter

Speos Sensor System Exporter (SSS Exporter) is a standalone tool to post-process exposure map calculated by Speos. By utilizing the camera sensor, Speos can generate an extended map known as the Spectral Exposure map, providing detailed information on the acquisition of camera sensors, expressed in Joules/m²/nm for each pixel. Exposure maps can be generated during dynamic or static inverse simulation by enabling the timeline parameter and specifying the integration time and trajectory file in Camera Sensor definition. SSS Exporter converts the exposure map into a Raw image, an electron map, and developed image, based on a Reduced Order Model of the sensor following EMVA 1288 standard (release 3.1). The electron map can be generated using the EQE data of the imager from Lumerical or from the imager’s datasheet. For this study, Lumerical simulated EQE data saved in a *.json file was used.

Run and Results

Instructions for running the model and discussion of key results

To explore the camera performance, we tested the system under different ambient illumination conditions as day and night with/without a lamp.

Step 1: Lens system design with Zemax OpticStudio - (not covered in this article)

The lens system is designed in Zemax OpticStudio and exported via “Export to Speos Lens System” tool in Zemax OpticStudio to be used by Speos.

If you want to explore the lens system which is designed in Zemax OpticStudio and its corresponding ROM (*.OPTdistortion file) , which can be used in Speos, you can find the relevant files located in the Input from Zemax Lens system folder.

More information about “Export to Speos Lens System” tool in Zemax OpticStudio can be found in Using the Export to Speos Lens System tool article.

Step 2: Speos Simulation

1. In Speos 2023 R1, open Speos simulation file Security-camera_demo.scdocx from CMOS Sensor Camera 2023R1 folder
2. Compute the 'Camera.Day' simulation. Note: the simulation time is about 15min with 16 physical cores.
3. Click on the 'Camera.Day' simulation in the simulation panel. The simulation results include an Exposure.xmp result file which is the Exposure map captured by the Camera sensor from the scene with Daylight source. Double-click the Exposure.xmp result to open the exposure map in the Virtual Photometric Lab.

   

4. Reproduce steps 2 to 3 for 'Camera.Night' and 'CameraNight.with.lamp' simulations.
5. Inside the Virtual Photometric Lab, change the 'True Color' to 'Black to white Color' to review the luminous exposure values at different positions.
6. To view the results in radiometric units instead of photometric units, you can simply select the ‘Radiometric Units’ option from the View menu in the XMP results. This will enable you to visualize the irradiance/exposure results in the appropriate radiometric units.
7. Click on the 'Level' tool to see the legend bar for the radiometric exposure value.
8. Click the 'Measures' tool and modify the area of interest, to measure the average exposure value on the desired area. You can also import the Full_Sensor_Area_Measurement_Template by navigating to the file menu and selecting "import template" in the XMP viewer from Speos Output files folder. This will enable you to accurately measure the exposure captured by the whole sensor area.

Result with Daytime source:
It shows the measured exposure value from the 3D scene during the day with a scale bar on the right side.

The measurement area of interest at the sensor can be defined using a different shape such as rectangular shape capturing the light from the scene to the sensor. In this example, the measurement area of interest covers the whole area of the sensor. Inside the measurement information table, the average exposure value captured by the whole area of the sensor with 0.05s integration time during the day is shown which is about 8 mJ/m².

The spectral exposure results of camera simulation also enable us to evaluate the camera optical properties such as distortion, vignetting, and resolution at various sensor locations.

Step 3: Lumerical Simulation - FDTD, CHARGE - (not covered in this article)

The coupled optoelectronic simulation of the CMOS imager is done in FDTD and CHARGE and the resulting EQE information is exported as a *.json file to be used for postprocessing.

More information about CMOS imager design in Lumerical FDTD and CHARGE can be found in Broadband Simulation of a CMOS Image Sensor – SPEOS Interoperability .

Step 4: Speos Sensor System Exporter

The content of the SSS exporter package is (a) SSS Exporter executable file. (b) Template of inputs and sensor files. (c) Template of *.bat file to start SSS Exporter executable. This *.bat file defines the path to the SSS Exporter exe and Input Yaml file.
Note: Yaml is a human-readable data serialization standard that can be used in conjunction with all programming languages and is often used to write configuration files. A Yaml file contains keys: - Each key can be associated to value(s).
The inputs and main parameters are saved in a file coded with Yaml standard with txt editable and contents instructions that SSS exporter can execute. The sensor properties (sensor properties, options and used conditions) used for post processing analysis are saved in Sensor Yaml coded file. The Post processing analysis done on the exposure map can be split into two main steps: (a)Exposure map to raw image and (b) Raw image to developed image. Calculations done in the first step are based on the EMVA 1288 standard with the possibility to improve the model using data generated from Lumerical model (quantum efficiency with angular dependency). Calculations done in the second step are based on a generic basic model.
The SSS Exporter can be started using a *.bat file and it comes with a minimal GUI that gives feedback about the calculation done.

Note: The SSSExporter.exe file can be found in the C:\Program Files\ANSYS Inc\v232\Optical Products\Viewers folder within the Speos 23R2 installation directory. If your local installation is not on this path, you need to adjust it in the batch file.

Note: If you obtain the message "Impossible to connect to the Virtual Photometric Lab", you need to run Virtual Photometric Lab 23R2 as administrator once.

1. To post process the Exposure maps from the Speos camera simulation, navigate to the input folder of the Speos Sensor System Exporter and save the Exposure map files there.
2. Open the input *.yaml file from Speos Sensor System Exporter folder and edit the Exposure maps key in Set 0, 1 and 2 to align with the name of the exposure maps (Day, night, and night with lamp respectively) you saved in the input folder.
   

   

3. Save the Lumerical EQE file (*.json) in the same input folder as the exposure maps.
4. Open the Sensor *.yaml files (Daytime.yaml and nighttime.yaml) from Input folder and establish a connection to the Lumerical EQE data by using the Lumerical data key. Modify the Lumerical data key to align it with the name of the recently saved *.json file located in the input folder.

   

5. Upon double-clicking the *.bat file, the associated electronic map, and final image will be promptly generated and stored in the designated output folder. If you want to generate the raw image and noise image, you can uncomment the corresponding keys, Raw Export and Noise Export, in the input yaml file.

Result with different lighting condition (Day, Night with and without lamp)
A comparison of electronic map results at the sensor under different lighting conditions (day, night with and without lamp) is shown below.

The following table presents a comparison of the average number of electrons obtained from the sensor under varying lighting conditions (day, night with and without lamp) for the measurement area of interest, which is the head of the person.

The current version of SSS Exporter enables users to measure Electrons in terms of their number from the electron map. To do this, follow these simple steps:

1. Open the *_ electron.xmp file from outputs folder of Speos Sensor System Exporter folder with XMP viewer.
2. Export the electron map as a *.txt file by selecting the Export tool from the File menu.
3. Edit the data on the second line in the text file by replacing the number 9 with 0 and saving the txt file. The data on the second line represents the unit Tag, where 9 is for unitless quantities and 0 is for irradiance.
4. Import the txt file using the Import tool in the XMP viewer and use the measurement tool to analyze the electron map in terms of the number of electrons per pixel on the desired area (Note: Ignore the units shown on your measurements).
5. To measure the number of electrons, present in the head area, you can either use Measure tool directly or make use of the pre-exported templates. To do this, go to the Speos Sensor System Exporter\Outputs folder and locate the E_Number_Measurement_Head_Area_Template. In the XMP viewer select "import template" from the file menu and Navigate to the E_Number_Measurement_Head_Area_Template and select it to import it into the XMP viewer. You can now click on the Measurement tool and read the corresponding data for your analysis.

Explanation of image quality metrics

The analysis of the electronic maps, raw and final images under varying illumination conditions, gives users the means to evaluate the camera's performance in optical and sensor design, and post-processing suitability.

For instance, the electronic map results indicate that the low number of electrons detected during the Night condition results in significant underexposure and a low-brightness final image. This could be attributed to the low light levels during night-time and/or poor quantum efficiency from the sensor, which can result in a low conversion efficiency of photons into electrons. The potential solutions could be to increase the exposure time during photometric camera simulation, use a lens with a larger aperture or increase the light level during the night (however, this may not be realistic.) Furthermore, by adjusting the postprocessing sensor definition and fine-tuning the operating condition gain factor and system gain factor, as well as ensuring sufficient Bits depth and Dynamic range, the performance of the resulting image can be improved. As part of our study, we have increased the operating condition gain factor in the sensor yaml file to 50 for Night illumination condition.

Based on the analysis, it can be concluded that this camera is not suitable for low-light conditions. Even with the addition of a lamp during nighttime, the resulting images are still underexposed. To obtain more realistic results with this camera, the user may adjust the 'operating gain' in the Sensor Yaml file to ensure the correct exposure during the Night condition, but caution must be taken to avoid overexposing the image during the Day condition.

In addition, the electron map can be used to explore if the performance of the designed electronics meets the requirement for the application based on different environmental conditions. For example, designers can input electronic maps into the full circuit simulation of readout electronics and analyze the full system performance.

Model Settings

Description of important objects and settings used in this model and instructions for updating the model based on your device parameters

SSS Exporter - Sensor Parameter:
Using a camera sensor, Speos can generate an exposure map in front of a CMOS sensor. This exposure map can then be converted into an electron map, noise map, raw image, and developed image using Ansys Speos post processing tool called SSS Exporter. To evaluate the postprocessing unit's performance, user can customize key values in the sensor YAML file, including the sensor's EMVA Standard data, preprocessing data such as Speckle removing filters, operating conditions, sensor EQE Lumerical data, and parameters used in development calculations, such as sensor linearization, rescaling, and colorimetry.

Camera simulation : In Speos there are 3 main types of camera models to generate an irradiance map or exposure map.

1. Basic camera sensor which is based on a pin-hole camera
2. Reduced Order Model (ROM) camera sensor which uses *.OPTdistortion data from the Zemax OpticStudio model also called Speos Lens System
3. Irradiance sensor with full lens system which uses full Zemax OpticStudio CAD model

Selecting between different camera simulations is a balance between accuracy and simulation time. The full lens system would be the most accurate one, but simulation with a basic camera model and ROM camera would be about 100 times faster. Usually, the ROM camera is the best balance between accuracy and simulation time for most of the analysis and is the one we use in this project.

The ROM camera sensor allows easy modification of its setting such as:

• Camera position
• Lighting transfer function exported from Zemax OpticStudio model (*.OPTdistortion file)

Camera Sensor position : In camera definition, the camera axis system, x and y direction are defined by an origin system (Camera_origin). You can move the ‘Camera origin’ using ’Move’ tool which is in ‘Design>Edit’ panel, vary camera position and direction to evaluate the performance of the design in other scenarios. In addition, you can create new origin using ‘origin’ tool in ‘Design>Create’ panel and set the camera axis to the new origin you created.

Camera Sensor Optical Distortion (Lighting transfer function):
In Camera definition, in ‘Distortion’ section, a file with *.OPTdistortion extension can be loaded which is editable as text file. This file has information regarding the object to image angular relationship and allows Speos to render the camera behavior.

[[NOTES:]] Depending on which version of *.OPTdistortion input file is used for camera optics, different information from the lens system could be included. Distortion curve V2 describes an enriched camera model which considers distortion, variable origin (Entrance Pupil Point), vignetting, resolution, and depth of field while Distortion curve V1 is the simplified version and has only the information regarding the chief ray angle curve. The file format of an example distortion curve V1 and V2 is shown below.

Further details about camera sensor settings and distortion curve can be found in Camera Sensor Distortion Curve (ansys.com) .

Angular Quantum Efficiency from Lumerical
The interaction of light with the sensor microstructure and the collection probability of photo-generated charged carriers is simulated in Lumerical FDTD and CHARGE, respectively. The final output of these simulations is the external quantum efficiency (EQE), which is the ratio of collected charge to incoming photons. The EQE data must be calculated for angles within the range of chief ray angles incident on the sensor (from the Zemax OpticStudio model) and stored in a *.json file to be used by the Speos Sensor System Exporter.
The effect of different configurations of the CMOS imager on the final image can be studied using SPEOS. For example, by placing a white uniform display source as a target, the uniformity and efficiency on different sensor channels based on different micro-lens designs of CMOS imager can be explored.

Optical Properties
To determine how rays interact with geometry in Speos, we need to define the optical properties of the objects included in the simulation. The scene of this example includes three main components:

• Surveillance_Camera: Camera, LED sources
• Building
• Person

In Speos and in the real world, there are two types of optical properties, surface/face optical properties such as optical polished, rough, coating and volume optical properties such as air, glass, and so on.

Different options are available to define optical properties files including user definition via optical property editors, Plugin for surface properties that is programmable by the user, and an optical library that contains a set of surface and material properties measured or defined by Ansys which can be downloaded in Add-On Packages on Ansys Customer Portal .

Light Source
In Speos there are different light sources available including artificial and natural lights. In this project, we set up a surface source as a lamp and a uniform ambient source as day and night which allows us to define a common luminance for the entire sky. The luminance of the entire sky during the day usually ranges from 1000 to 20000 cd/m² and in this project we used 7000 cd/m2 for the day and 25 cd/m2 for the night. Also, the source spectrum can be selected from Library, or using Blackbody to set the temperature of the source spectrum in Kelvins. In this example, we used Blackbody and set daylight at 6500 K and night at 5000 K. More information regarding surface and ambient sources can be found in Surface Source (ansys.com) and Ambient Sources (ansys.com).

Simulation Settings
Different types of simulation are provided in Speos such as direct and inverse simulation. With camera models, we should use inverse simulation where light rays propagate from the sensor to the source.

Simulation Meshing Settings: Meshing settings are critical for getting the correct simulation results. Meshing settings define the quality of geometries that will be simulated. The finer mesh gives better results but also requires longer simulation time. Rough mesh can lead to poor results, especially at the precise optical component. Meshing in this project is set to be proportional to the body size. Further details about mesh settings can be found in Meshing Properties .

Stop Conditions/Number of Passes : In an inverse simulation, you can define the criteria to reach for the simulation to end which is the stop condition.
To stop the simulation after a certain number of rays were sent, in the inverse simulation definition panel, you can set the 'number of passes limit' to True and define the number of passes. The number of passes is the number of rays per pixel per wavelength; so, the total number of rays propagated in the system is: number of passes * number of pixels * number of samplings of spectral data. In this project, we used 50 passes to achieve a good result without too much noise.
If you want to adjust the inverse simulation advanced settings, see Adjusting Inverse Simulation Settings.

Additional Resources

Additional documentation, examples and training material

Relevant Speos Ansys Learning Hub courses:

• Speos Getting Started
• Speos Workflow & Geometry Management
• Speos Camera Sensor