CMOS Sensor Camera - Image Quality Analysis in a 3D Scene

In this example, we introduce a simulation workflow to analyze the image quality of a camera system from a combination of an optical system and a CMOS imager within a specific environment with different lighting conditions. This example covers mainly the Ansys Speos part of the overall workflow. The optical system is designed with Ansys Zemax OpticStudio and exported to Ansys Speos for system-level analysis. The CMOS imager is designed with Ansys Lumerical and exported to Ansys Speos.

download example

Overview

Understand the simulation workflow and key results

In camera systems, CMOS (complementary metal oxide semiconductor) imager is an electronic component where incident absorbed photons generate a photocurrent that can be digitally processed. In this example, we used Ansys complete optical solution and imported the optical system information from Zemax OpticStudio and CMOS imager from Lumerical into Speos to do a full camera system analysis in a 3D scene and simulate the electronic map generated by the imager. This interoperability workflow takes into consideration the interplay between the macroscopic camera lens and the microstructure of the CMOS image sensor when simulating the entire optical system. With the Speos capabilities of handling realistic illumination and photometric/radiometric physics-based rendering, users can easily optimize the components and construct an accurate view of the final electronic map as recorded by the image sensor to design application-based cameras.

There are four main tools needed for this virtual solution :

1. Zemax OpticStudio and/or Speos Lens System Importer (external tool) to export lens model designed in Zemax OpticStudio to be used by Speos
2. Speos to generate spectral irradiance map in front of the CMOS imager
3. Lumerical FDTD (Finite Difference Time Domain) and CHARGE to compute the quantum efficiency of the sensor as a function of incidence angle and wavelength
4. Speos Sensor System Exporter to combine Speos and Lumerical results and generate the electronic map captured by the sensor

As this example requires Lumerical, Zemax OpticStudio and Speos simulation, we mainly limited ourselves to Speos simulations here as described by the following steps:

Step 1: Lens system design with Zemax OpticStudio (not covered in this article)

In this step, we design the camera lens system in Zemax OpticStudio and export the lighting transfer function of this lens system and format it into a readable file by Speos(*.OPTdistortion). The reduced order model (ROM) generated lens model is called Speos Lens System (SLS). This operation is done once for the lens system which enables us to perform fast ray tracing camera simulations in Speos afterward.

Here we use a compact and efficient lens system for mobile phone cameras designed with Zemax OpticStudio. The incident angle at the sensor surface increases to 35 degrees at about 2 mm distance from the center of the sensor and then decreases to 25 degrees at the corner of the sensor.

SLS can be generated by Zemax OpticStudio (for symmetric lens system), or by Speos Lens System Importer -SLS importer- (for asymmetric lens system), or from Chief ray angle curve information from a datasheet (if the lens design information is not available).

Step 2: Speos Simulation

The *.OPTdistortion file is imported into the Speos camera sensor to define the lens performance of the camera system and evaluate sensor perception in a 3D scene with realistic illumination conditions. We run a ray tracing photometric ROM camera simulation which is about 100x faster than the full lens system simulation in Speos and extract the key imaging metrics such as spectral irradiance map. The irradiance map in front of the CMOS imager sensor is calculated from the full 3D scene shown below under different ambient lighting conditions including day, night, and night with a lamp.

Step 3: Lumerical Simulation - FDTD, CHARGE- (not covered in this article)

The spectral irradiance map simulated by Speos in front of the CMOS imager needs to be combined with the sensor’s quantum efficiency to generate the raw electronic map. Lumerical FDTD and CHARGE tools have been used to quantify the quantum efficiency of the designed CMOS sensor. The CMOS image sensor is composed of microscopic pixels with optical and electronic elements. The main optical elements are micro-lenses and color filters used to focus light of the desired wavelength on the correct spot in the silicon substrate at the bottom of the imager. The absorbed photons generate charged carriers that are collected and transferred for detection on the electronic side. The electronics have components including gates and interconnects which can interfere with the path of light inside the sensor. The coupled optoelectronic simulation is done in FDTD and CHARGE.

Step 4: Speos Sensor System Exporter

Speos Sensor System Exporter is a tool used for post-processing the irradiance map captured by the camera sensor in Speos. To generate the raw image recorded by the sensor in terms of the number of electrons or current collected at each pixel, we use the Speos Sensor System Exporter tool to combine the spectral irradiance from Speos and the EQE data of the imager from Lumerical. This tool first calculates the exposure map of the Speos simulation from the product of the irradiance map and the camera integration time, which is an editable parameter in the EQE *.json file. Then from the EQE data, this tool calculates the responsivity, which is the rate of collected charge per incident power and multiplies responsivity by spectral exposure map to generate the electronic map.

Run and Results

Instructions for running the model and discussion of key results

To explore the camera performance, we tested the system under different ambient illumination conditions as day and night with/without a lamp.

Step 1: Lens system design with Zemax OpticStudio

The lens system is designed in Zemax OpticStudio and exported to be used by Speos.

Step 2: Speos Simulation

1. In Speos 2022 R2, open Speos simulation file Security-camera_demo.scdocx from CMOS Sensor Camera 2022R2 -Post Processing Steps folder
2. Compute the 'Daytime_Camera.8Mp.Day' simulation. Note: the simulation time is about 1h with 8 cores.
3. Click on the 'Daytime_Camera.8Mp.Day' simulation in the simulation panel. The simulation results include an Irradiance.xmp result file which is the irradiance map captured by the Camera.8Mp sensor from the scene with Daylight source.
4. Double-click the Irradiance.xmp result to open the irradiance map in the Virtual Photometric Lab
5. Inside the Virtual Photometric Lab, change the 'True Color' to 'Black to white Color' to review the illuminance values at different positions.
6. Click on the 'Level' tool to see the legend bar for the illuminance value.
7. Click the 'Measures' tool and modify the area of interest, to measure the average luminance value on the desired area.
8. Reproduce steps 2 to 7 for '*_Camera.8Mp.Night' and '*_Camera.8Mp.Night.with.lamp' simulations.

Result with Daytime source:
It shows the measured illuminance value from the 3D scene during the day with a scale bar on the right side.

A rectangular shape defines the measurement area of interest at the sensor and captures the light from the scene to the sensor during the day. Inside the measurement information table, the average illuminance value captured by the whole area of the sensor is shown which is about 54.6 lx.

The spectral irradiance results of camera simulation also enable us to explore the camera optical properties such as distortion, vignetting, and resolution at various sensor locations.

Step 3: Lumerical Simulation - FDTD, CHARGE

The coupled optoelectronic simulation of the CMOS imager is done in FDTD and CHARGE and the resulting EQE information is exported as a *.json file to be used for postprocessing.

More information about CMOS imager design in Lumerical FDTD and CHARGE can be found in Broadband Simulation of a CMOS Image Sensor – SPEOS Interoperability .

Step 4: Speos Sensor System Exporter

1. From SPEOS_LENS_IMPORTER_2022R2_WINX64 folder, use the link to download and run Speos lens system Importer.exe which is the tool that allows us to generate the electronic map by combining the irradiance maps (*.Irradiance.xmp ) we analyzed from Speos simulation in previous steps and the EQE file (*.json) from Lumerical.
2. In the editor panel of Speos lens system importer tool, write ‘opp’ and click ‘enter’ to activate the Output Pupil Position Calculation in SLS

   

3. Click on ‘Post-process’ icon, a Spectral Irradiance map to RGB map window opens. Select irradiance map collected by the camera with the day source and the EQE information exported from Lumerical for ‘’DATA From Speos’’ and ‘’DATA From FDTD’’, respectively. By clicking on ‘Start Post processing’ icon, the corresponding electronic and electronic_per_second maps will be generated and stored in the output folder. Note: The post processing step might take several minutes.
4. To evaluate electronic map results, click on the *_electronic.xmp result, the electronic map will be opened with the Speos Virtual Photometric Lab, basic lab .
5. In Speos Lab, click the 'Measures' tool and modify the area of interest, to measure the average number of electrons value on the desired area (the quantity is unitless so ignore the W/m2)
6. Repeat steps 1 and 5 for the irradiance map collected by night source with  and without lamp to review the average number of electrons value on Night and Night with lamp cases.

Result with different lighting condition (Day, Night with and without lamp)
It shows the comparison of the electronic map captured by the CMOS sensor with 20ms Integration time and full well capacity of 15,000 e- under different lighting conditions (day, night with and without lamp).

A comparison of electronic value results at the sensor under different lighting conditions (day, night with and without lamp) is shown in the table below. The measurement area of interest (head of the person) is identically defined on the electronic map.

The electronic map can then be used for further post-processing by the user.

Explanation of image quality metrics
The electron map can be used to explore if the performance of the designed electronics meets the requirement for the application based on different environmental conditions. As shown in this example, with different lighting conditions the number of electrons decreases drastically between night and day.
Also, electron maps allow the designer to check if the full well capacity (FWC) of the imager has exceeded; in this example, we have considered the full well capacity of 15,000 e- (which is an editable variable in EQE json file) above which saturation happens.
In addition, designers can input electronic maps into the full circuit simulation of readout electronic s and analyze the full system performance.

Model Settings

Description of important objects and settings used in this model and instructions for updating the model based on your device parameters

Camera simulation : In Speos there are 3 main types of camera models to generate an irradiance map

1. Basic camera sensor which is based on a pin-hole camera
2. Reduced Order Model (ROM) camera sensor which uses *.OPTdistortion data from the Zemax OpticStudio model also called Speos Lens System
3. Irradiance sensor with full lens system which uses full OpticStudio CAD model

Selecting between different camera simulations is a balance between accuracy and simulation time. The full lens system would be the most accurate one, but simulation with a basic camera model and ROM camera would be about 100 times faster. Usually, the ROM camera is the best balance between accuracy and simulation time for most of the analysis and is the one we use in this project.

The ROM camera sensor allows easy modification of its setting such as:

• Camera position
• Lighting transfer function exported from Zemax OpticStudio model (*.OPTdistortion file)

Camera Sensor position : In camera definition, the camera axis system, x and y direction are defined by an origin system (Camera_origin). You can move the ‘Camera origin’ using ’Move’ tool which is in ‘Design>Edit’ panel, vary camera position and direction to evaluate the performance of the design in other scenarios. In addition, you can create new origin using ‘origin’ tool in ‘Design>Create’ panel and set the camera axis to the new origin you created.

Camera Sensor Optical Distortion (Lighting transfer function):
In Camera definition, in ‘Distortion’ section, a file with *.OPTdistortion extension can be loaded which is editable as text file. This file has information regarding the object to image angular relationship and allows Speos to render the camera behavior.

[[NOTES:]] Depending on which version of *.OPTdistortion input file is used for camera optics, different information from the lens system could be included. Distortion curve V2 describes an enriched camera model which considers distortion asymmetry, variable origin (Entrance Pupil Point), vignetting, resolution, and depth of field while Distortion curve V1 is the simplified version and has only the information regarding the chief ray angle curve. The file format of an example distortion curve V1 and V2 is shown below.

Further details about camera sensor settings and distortion curve can be found in Camera Sensor Distortion Curve (ansys.com) .

Angular Quantum Efficiency from Lumerical
The interaction of light with the sensor microstructure and the collection probability of photo-generated charged carriers is simulated in Lumerical FDTD and CHARGE, respectively. The final output of these simulations is the external quantum efficiency (EQE), which is the ratio of collected charge to incoming photons. The EQE data must be calculated for angles within the range of chief ray angles incident on the sensor (from the Zemax OpticStudio model) and stored in a *.json file to be used by the Speos Sensor System Exporter.
The effect of different configurations of the CMOS imager on the final image can be studied using SPEOS. For example, by placing a white uniform display source as a target, the uniformity and efficiency on different sensor channels based on different micro-lens designs of CMOS imager can be explored. In addition, a detailed colorimetric study can be done to check the color variation at various positions on the sensor.

Optical Properties
To determine how rays interact with geometry in Speos, we need to define the optical properties of the objects included in the simulation. The scene of this example includes three main components

• Surveillance_Camera: Camera, LED sources - see Surveillance_camera.scdocx
• Building: -see Building.scdocx
• Person -see person.scdocx

In Speos and in the real world, there are two types of optical properties, surface/face optical properties such as optical polished, rough, coating and volume optical properties such as air, glass, and so on.

Different options are available to define optical properties files including user definition via optical property editors, Plugin for surface properties that is programmable by the user, and an optical library that contains a set of surface and material properties measured or defined by Ansys which can be downloaded in Add-On Packages on Ansys Customer Portal .

Light Source
In Speos there are different light sources available including artificial and natural lights. In this project, we set up a surface source as a lamp and a uniform ambient source as day and night which allows us to define a common luminance for the entire sky. The luminance of the entire sky during the day usually ranges from 1000 to 20000 cd/m² and in this project we used 7000 cd/m2 for the day source and 25 cd/m2 for the night. Also, the source spectrum can be selected from Library, or using Blackbody to set the temperature of the source spectrum in Kelvins. In this example, we used Blackbody and set daylight at 6500 K and night at 5000 K. More information regarding surface and ambient sources can be found in Surface Source (ansys.com) and Ambient Sources (ansys.com).

Simulation Settings
Different types of simulation are provided in Speos such as direct and inverse simulation. With camera models, we should use inverse simulation where light rays propagate from the sensor to the source.

Simulation Meshing Settings: Meshing settings are critical for getting the correct simulation results. Meshing settings define the quality of geometries that will be simulated. The finer mesh gives better results but also requires longer simulation time. Rough mesh can lead to poor results, especially at the precise optical component. Meshing in this project is set to be proportional to the body size. Further details about mesh settings can be found in Meshing Properties .

Stop Conditions/Number of Passes : In an inverse simulation, you can define the criteria to reach for the simulation to end which is the stop condition.
To stop the simulation after a certain number of rays were sent, in the inverse simulation definition panel, you can set the 'number of passes limit' to True and define the number of passes. The number of passes is the number of rays per pixel per wavelength; so, the total number of rays propagated in the system is: number of passes * number of pixels * number of samplings of spectral data. In this project, we used 50 passes to achieve a good result without too much noise.
If you want to adjust the inverse simulation advanced settings, see Adjusting Inverse Simulation Settings.

Additional Resources

Additional documentation, examples and training material

Relevant Speos Ansys Learning Hub courses:

• Speos Getting Started
• Speos Workflow & Geometry Management
• Speos Camera Sensor