Augmented reality systems provide good demonstrations of multiple ray paths and free-form optics. This article demonstrates how to set up an optical system for a HMD in Sequential Mode, using a wedge-shaped prism and freeform surfaces. It contains three OpticStudio files representing different stages of the tutorial as references.
Authored By Natalie Pastuszka
Introduction
Optical see-through head-mounted displays (OST-HMD) utilize augmented reality (AR) by optimizing two optical paths: the microdisplay projection imaging path and the see-through path. This is because AR overlays graphics onto the user's real environment, rather than wholly replacing it. This is extremely useful in a variety of applications ranging from aiding surgeons to mixed reality gaming.
Considering the application, it is important to design a compact and non-intrusive system with a wide field of view and low f-number. This article describes how to model such a system using a freeform wedge-shaped prism and a cemented auxiliary lens.
Patent reference
This basic idea for the design was referenced from Patent US 2014/0009845 A1. The optical system is largely dependent on tilt and decentering multiple surfaces. The system is comprised of a freeform surface (FFS) prism and cemented auxiliary lens (highlighted in yellow). This is because freeform surfaces allow for more degrees of freedom, allowing the use of fewer elements which results in a lighter system. The auxiliary lens is used to correct distortion and improve the see-through capabilities of the optical system.
Image is taken from patent and modified:
Design strategy
The OST-HMD consists of two elements: 1) wedge-shaped FFS prism and 2) cemented auxiliary lens. The FFS prism is designed first and optimized for pre-defined specifications for the microdisplay projection imaging path (first optical path). Afterward, the auxiliary lens is added using multi-configuration mode with the intention of minimizing distortion and eliminating optical power in the see-through capabilities of the system (second optical path).
The system is modeled “in reverse” to how the setup works in reality. In reality (in the physical system), the source for the HMD is the microdisplay and the image plane will be the retina of the human eye (the exit/entrance pupils of the HMD system and the human eye will be collocated). However, to model this setup accurately and optimize efficiently in OpticStudio, the system is defined in such a way that the exit pupil of the physical system is the entrance pupil as modeled in OpticStudio and the microdisplay is treated as the “image plane” of the system. Therefore, any rays referenced in the following article are described by the way they are modeled in OpticStudio.
To model this HMD, we will first set up the prism by inserting one surface at a time and tracing a single chief ray at one field angle through the system. We will insert Coordinate Break surfaces to tilt the prism faces and reflect the ray as needed. With this, we will need to consider the geometry of the design to properly define the Material for each surface. Due to the nature of Sequential Mode, we will simulate total internal reflection (TIR) by using an overlapped surface defined as a mirror and with Pickup solves. After optimizing the system for one field, we will use the Multi-Configuration Editor (MCE) to create a second path of the design, and we will then optimize the entire system for performance specifications, considering manufacturing means.
Wedge-shaped FFS prism
The initial setup consists of defining the entrance pupil at 6mm (with the intention of working it up, human eye pupil is 2-8mm) and a single field point while setting up all the surfaces to allow for a simpler ray trace. Once these surfaces are set up and the ray is traced accurately through the prism, the FOV and entrance pupil can be expanded progressively (further discussed in the section “Defining the field of view (FOV)” further in the article).
A prism designed in Sequential Mode is built from multiple individual surfaces that are tilted and decentered in such a way as to give the prism a shape. With this, it is necessary to consider how the rays travel through the prism to determine when and where a surface is necessary. Figure 2 below shows the direction in which the rays are traveling for modeling purposes, as well as the surface numbers.
The enlarged red numbers indicate the actual surface number as modeled in the Lens Data Editor (LDE), considering all surfaces have Coordinate Breaks associated with them. The black “S#” indicates the physical surface of the prism, as defined by the patent (these are also the numbers indicated in the Comments column).
For example, 8-9-10 is Surface 9 as listed in the final LDE (outlined in bolded red box) and corresponds to surface S1’ of the physical prism (the reflective/inside surface of S1). The smaller 8 and 10 correspond to surfaces 8 and 10 in the LDE and are used as Coordinate Breaks (dummy surfaces to tilt/decenter) for surface 9.
When modeling this, the stop (entrance pupil of modeled system) is made as the Global Coordinate Reference. The first surface of the prism (S1 or Surface 3 in LDE after Coordinate Breaks are inserted) is initially placed 18.25mm (Thickness of Surface 1 in LDE) after the stop of the system to serve as the eye relief. The tilt/decenter should be used as necessary to eventually aim the rays to the image plane. A rough idea of the amount of tilt/decenter per element can be taken from the images and tables in the patent.
Rays are transmitted through surface S1 (Surface 3 in LDE) to S2 (placed in LDE after the Coordinate Break return of S1, namely as Surface 5 in LDE which becomes Surface 6 with the use of Coordinate Breaks). This surface is indicated as a mirror in order to reflect the rays within the prism. In reality, as described in the patent, this surface is coated to be a half-mirror to allow the rays to serve both paths: reflected within the prism to the microdisplay and transmit light as part of the see-through path. This see-through setup will be modeled in a multi-configuration mode later in this article (Section Setting up Multi-Configuration Editor). The tilt/decenter tool is used once again to tilt the surface according to the figure in the pattern. The surface now becomes Surface 6 with the Coordinate Breaks in place.
Note: To give a better visual of where the rays are tracing, the microdisplay image plane is set up and tilted/decentered approximately to that which is in the patent. In this way, there is a visual for where the rays should be aiming, provided the geometry is set up properly.
Analyzing the layout, the rays next need to reflect from the inside of S1 (namely, S1’) to reach the microdisplay/image plane. According to the patent, these rays reflect off this surface due to total internal reflection (TIR), as is further discussed in the “Total Internal Reflection (TIR)” section of this article.
TIR is not supported in Sequential Mode and therefore this reflection needs to be improvised by making S1’ a mirror. Pickup Solves are used on all parameters in order to mimic S1 (Surface 3 in LDE) since physically this is the same surface; it is necessary to define it as two separate surfaces because of the ray-tracing capabilities in Sequential Mode and the change in transmissive properties of the surface (transmits light as S1, but reflects light due to TIR as S1’ in the physical system). The LDE snippet below shows the parameters that were picked up by Surface 9 (other parameters such as Decenter X or Decenter about Y would also utilize pickups from Surface 3 if it would be desirable to have those decenters in the design).
The last surface before the image plane (S3, Surface 12 in LDE) is added to the model in the same way as the previous surfaces, with the necessary amounts of x-tilt and y-decenter introduced.
Defining field of view (FOV)
The FOV in the system needs to be defined by as many points as possible, due to the freeform surfaces varying more with field. In this way, OpticStudio is able to optimize more effectively for the intermediary fields between those explicitly listed in the System Explorer. Similarly, the field points need to be defined both in the x and y directions. Since the system is not rotationally symmetric (symmetric about the YZ plane, but not the XZ plane), we are no longer able to assume that the rays will behave in the same manner in both the positive and negative directions.
Total internal reflection (TIR)
In the physical diagram, rays emitted by the microdisplay reflect off S1’ by means of TIR. This phenomenon where light is completely reflected back into its medium of propagation occurs when light that is traveling inside a higher-index material strikes a surface boundary of a lower-index medium at an angle greater than its critical angle. The critical angle is defined as follows:
where nr is the index of the material where refraction would occur and ni is the index of the incident medium. In this case, the TIR condition is satisfied when rays traveling within PMMA (n= 1.492) strike surface S1’ at an angle greater than θc= 42.09 degrees. The real ray angle constraints were used in this optimization process.
Creating user-defined rectangular apertures
The size of the surfaces can be defined by adjusting the semi-diameter as well as the apertures placed on the surfaces. This particular system will have rectangular apertures due to the physical geometry of the wedge-shaped prism.
The rectangular apertures are defined for every surface modeled in the prism (not the Coordinate Breaks, since they only serve as “dummy surfaces” and aren’t a representation of where the rays strike). The apertures are set by clicking on the particular surface and extending the Surface Properties dialog. Under the Aperture Tab, the Aperture Type is selected to be a Rectangular Aperture. The available half-width and decenter parameters are adjusted to shape the given surface and only use as much of the surface as is desired (useful for practicality purposes and keeping the system compact).
Optimization
The system is optimized for an RMS wavefront centroid with the rings and arms being increased as the design improves. The constraints are added gradually as needed using the Merit Function Editor. The main constraints include a range for effective focal length (EFL), thicknesses, global coordinates, ray path lengths, tilt/decenter parameters, angles, and distortion.
It is pertinent to align surface S1’ to S1 (Surfaces 9 and 3, respectively, in the LDE) using the global coordinate constraints (GLCZ/GLCY/GLCZ), since physically this is just one surface; it is simply modeled as two surfaces due to the nature and capabilities of Sequential Mode and the difference in optical properties as the rays travel through the prism. These operands are used in addition to the already established pickup solves for surfaces 9 and 3 (as mentioned earlier).
The ray path length constraints were approximated as necessary, specifically from S3 to the image plane and S1’ to the image, to keep a general location of the image plane to maintain compactness of the system (for practical reasons).
Similarly, the tilt/decenter parameters are constrained to maintain a general shape of the prism and prevent the surfaces from deviating too strongly from each other.
The initial surface type for each surface is a Standard Lens, and is slowly moved up to an Even Asphere, and some eventually a freeform surface; in this case, an Extended Polynomial surface was used with varying for surfaces S1 (and S1’ through pickups) and S3 (Surfaces 3, 9, and 12 in LDE, respectively). Surface 2 remained an Even Asphere.
Setting Up the Multi-Configuration Editor
Up to this point, the first optical path (projection imaging path) has been optimized with the FFS prism. The second optical path of the system (see-through path) needs to be configured and optimized, primarily through use of a second element in the OST-HMD system: an auxiliary lens which is cemented to the S2 surface of the FFS prism.
The optimized FFS prism (with rectangular apertures) and its respective LDE are as follows below (values slightly differ in final version due to optimization and addition of fields). The aspheric and freeform coefficients can be obtained in the final sample file.
Note: the color of the rows has been changed just out of personal preference and personal clarity when optimizing.
To set up the entire OST-HMD system, surface S4 (shown below) of the auxiliary lens needs to be added to the already existing FFS prism. The Multi-Configuration Editor (MCE) will then be used to split the entire system into two configurations based on the optical paths described at the beginning of the article: the projection imaging path (optical path 1) and the see-through path (optical path 2). The images below show the result of both configurations overlaid on one another (top) and each separate configuration (bottom):
To set up multi-configuration mode, the second optical path is considered and therefore S4 is added right before the image plane (S4 is the last surface rays will transmit through before the image in optical path 2). The initial LDE with the added S4 (rows highlighted in blue) and its respective Coordinate Breaks is demonstrated below.
The optical system now needs to be split into two individual path lengths via the Multi-Configuration Editor (MCE). By dividing the system into two separate configurations, we are able to optimize each system for different criteria and with respect to different surface properties. Namely, we can further optimize our FFS prism with a projection display for RMS wavefront for a finite image plane, while optimizing the auxiliary lens for the see-through path which is afocal in image space. In this see-through path, we want to be able to view the environment just as we would with regular protective goggles (hence afocal in image space); this requires imaging rays from “infinity” onto our cornea and optimizing the auxiliary lens to provide minimal distortion and no optical power through the see-through path. There are various Help Files available in OpticStudio that aide in setting up the MCE (files “Multiple Configuration Editor” and “Multi-Configuration Operands” are especially useful).
Looking at the individual configuration schematics and the expanded LDE, we determine that Configuration 1 will be compromised of Surfaces 0-13 of the LDE, as well as the image plane (surfaces 17 and 18 of LDE). Configuration 2 consists of surfaces S1 and S2, but no longer utilizes surfaces S1’ or S3. Therefore, those surfaces (and their respective Coordinate Breaks) in the LDE are omitted. Configuration 2 is set up to only consider Surfaces 0-7, 14-16 and 18 of the LDE; the Coordinate Break for the image surface is omitted since we want the image plane to be perpendicular to the z-axis, as if to model light from the environment.
Looking at the MCE, in the configuration columns, a value of “1” indicates that the operand is active and “0” indicating the operand is inactive/not utilized for the particular configuration. The omitted surfaces in each configuration are “ignored” via the IGNR operand in the MCE, where the surface to be affected in a configuration is specified. For example, looking at rows 3-5 of the MCE, surfaces 14-16 are indicated to be ignored for configuration 1, respectively. It is important to note the use of the GLSS operand to change the material type of Surface 6 in the LDE (S2 of the FFS prism); this surface was originally indicated as a mirror (for the projection imaging path, Configuration 1) in order to model the reflective properties of the actually half-mirrored surface. When modeling the see-through path (Configuration 2), the transmissive properties of the surface are modeled in Sequential Mode by means of indicating the material choice for the auxiliary lens. The AFOC Multi-Configuration Operand serves to activate the “Afocal Image Space” option for Configuration 2 (see-through path).
To optimize a multi-configuration system, both the parameters in the MCE and in the Lens Data Editor can be varied during optimization. The MCE is updated to include Configuration 2 through the “CONF” operand, with operands listed under “CONF” applying only to the configuration as specified under “Cfg#” (until the next “CONF” operand).
Analysis
The system can be analyzed for performance specifications by either using the MCE or the Tools featured in OpticStudio. Among some of the more relevant analyses, the Huygens PSF is utilized for this particular system due to the freeform nature of the system as well as the lack of rotational symmetry (for more information on which MTF to use, see "Why are FFT and Huygens MTF results different on tilted image surfaces?"). The sag and curvature of the surfaces can be taken into consideration through use of the Sag and Curvature selections in the Menu Bar under Analyze...Surface...Sag/Curvature, and utilizing the Settings dropdown menu on the plot to specify further details.
Another important tool to consider is the Field Map under Analyze...PAL/Freeform. Using the Setting drop down menu on the plot, we are able to analyze various features, including the power incident on a given surface. In this example, we can use this feature to analyze the amount of power which reaches the image plane in the see-through path of our design. Due to the application of our system, the see-through path should have minimal power (about less than 0.5D) so the human eye cannot perceive this discrepancy, which can cause fatigue and headache among other complications.
References
1. Cheng, Dewen, Hong Hua, and Yongtian Wang. “Optical See- Through Free- Form Head- Mounted Display.” U.S. Patent 0009845. 9 January 2014
KA-01334