Introduction
CML Compiler's Quality Assurance (QA) testing platform supports both Standard and Custom QA tests to help users verify that models are behaving as expected. Standard QA tests are provided with CML Compiler, and are used to check that the model is implementation is successful, for the input data provided. Custom QA tests provide users the flexibility to design tests with:
-
- Custom test benches, saved in INTERCONNECT project files or written by script
- Custom results extraction and analysis
- Comparison between simulation results and measurement data loaded from file
- Custom error reporting on key figures of merits and results
- Custom plotting
- etc.
Note: Custom QA tests are currently available for INTERCONNECT CMLs only |
How to create a custom QA test
Two steps are required for adding custom QA tests for an element.
- Create custom QA test script file(s) in source folder (for more information see below)
- Add list of script file names to element .json data file:
Variable Type Description custom_qa_filenames Cell Cell of size N, where N is the number of custom QA script files.
Each element of the cell is a string containing a script file name.
Note: For an example, please see the gc_strip_te_c template files. |
When CML Compiler is run with the “all” argument, an <element_name>_INTC_custom_test.lsf file is automatically generated in the element source folder. This file serves as the element's single top-level custom test file to run, as it configures the test environment and runs multiple custom QA tests. It executes the following:
- Changes the directory to the source folder
- Loads the <element_name>.json source file. This way your source data is loaded in the workspace and available to use in the QA test.
- Adds useful variables to the workspace (see below)
- Runs the script files listed in the custom_qa_filenames variable of the element source data
Variable | Type | Description |
---|---|---|
foundry_element | String |
Name of element |
library_name | String | Name of library |
source_directory | String |
Directory path of the element source data |
foundry_directory | String | Directory path of the library |
process_prefix | String |
Process prefix defined in the XML file. This is an empty string when no prefix is defined. |
model | String |
Model name of the element, as it appears in the CML, including both the process prefix and element name |
Custom QA test script
The custom QA test script has key pieces:
- Test benches: Set-up and run circuit simulations to obtain results.
- Results analysis and error reporting: Analyze results and determine if tests pass or fail.
- Plotting (optional): Plot relevant results.
Test benches
The custom QA test script can be used to either load a test bench from an INTERCONNECT project file, or it can create the test bench by script.
A test bench can be loaded from an existing project file using the load command. It is recommended to use the full file path, or use the "source_directory" variable in the workspace. It is also recommended to use the switchtolayout command to ensure that file can later be run for up-to-date results. Test bench elements can be updated based on input data for calibration. It is then necessary to run the simulation to obtain results.
The following is an example showing a test bench loaded from file, edited based on element data and run:
load(source_directory + "/"+"grating_qa.icp");
switchtolayout;
setnamed("ONA_1","center frequency",peak_wavelength);
run;
Alternatively, the test bench can be written by script. It is recommended to start from a blank project using the new command. The addelement command is used to include an element from your library. To ensure the correct element is selected, it is recommended to use workspace variables "design_kit_name" and "model". The test bench can be build around this element using Primitive elements, including analyzers, sources and detectors. For ideas on building test benches, please see the Standard QA test files that are generated by CML Compiler in the "source/<element name>/QA/" folder.
The following is an example where the test bench is composed of the test element connected to an Optical Network Analyzer:
new;
addelement(design_kit_name + model);
set("name","test element");
addelement("Optical Network Analyzer");
set("name","ONA_1");
setnamed("ONA_1","y position", 100);
connect("ONA_1","output","test element","opt_1");
connect("ONA_1","input","test element","opt_2");
run;
Results analysis and error reporting
Once the simulation is run, results are extracted from the analyzers in the circuit using commands like getresult, getattribute. To compare the simulation results and measurement results, external data can be loaded from file using commands like jsonload, matlabload, readdata.
To register a test pass/fail, use the "assert" command.
Syntax |
Description |
---|---|
assert(msg,pass); |
If pass = false, prints an Error to the Script Prompt with error message msg. If pass = true, no error is reported. |
The following is an example of code used in a custom QA test for a grating coupler, where the simulated peak wavelength of the spectrum (peak_wavelength_sim) is compared against a reference value (peak_wavelength_ref). The test passes if the two values agree to within the wavelength_tolerance.
msg = "Peak wavelength has been changed by: " + num2str(abs(peak_wavelength_sim-peak_wavelength_ref));
pass = abs(peak_wavelength_sim-peak_wavelength_ref) <= wavelength_tolerance;
msg_result = "Peak wavelength is " + num2str(peak_wavelength_sim*1e9) + " nm, and should be " + num2str(peak_wavelength_ref*1e9) + " nm.";
if(!pass) {
? "ERROR - " + msg_result;
} else {
?"Passed! - " + msg_result;
}
assert(msg,pass);
Note: Your custom QA tests should include at least one assert command. This indicates the pass/fail of the element test. If no assert commands are included, cml-compiler runtests --custom
will report "NO TESTS FOUND". |