This article provides the procedure for running QA tests generated by CML Compiler. It assumes that a compact model library has already been generated and that the QA tests have been generated by running the tool with either the "all" or "test" arguments.
If an INTERCONNECT CML has been generated using the "library" argument, the corresponding QA tests can be generated by navigating to the foundry directory and running:
cml-compiler test
For Verilog-A CMLs use "--veriloga" argument as discussed in Getting Started with CML Compiler.
For a list of command options, use the --help option like so:
cml-compiler test --help
After QA tests are generated they can be run automatically through CLI (Command Line Interface) for INTERCONNECT CMLs and Verilog-A CMLs, or they can be run manually as discussed in Running QA Tests Manually.
Note: Custom QA tests are only generated using the "cml-compiler all" command, not "cml-compiler test" |
Running QA Tests through CLI for INTERCONNECT CMLs
Using the runtests argument
To execute all of the QA tests for all INTERCONNECT models, navigate to the Foundry directory and execute the following command:
cml-compiler runtests
The runtests argument for the cml-compiler shell tool is most useful for running large numbers of tests automatically. It functions by using the XML Data file as an element index, which is used to locate the QA test scripts (*_INTC_test.lsf) for each element. Running _INTC_test.lsf script, runs all the test-bench scripts of the specified element.
To specify a subset of INTERCONNECT models to run QA tests for, use the --element_list option, like follows:
cml-compiler runtests --element_list element_1 element_2...
where element_1 and element_2 are element names within the CML. Using this command allows the execution of specific element QA tests which are included in the compact model library (.cml file) and XML data file.
To specify certain test-bench scripts to run, use the –test_bench option, like follows:
cml-compiler runtests --test_bench keyword_1 keyword_2...
Where keyword_1 and keyword_2 are the Figures of Merit (FOMs) that can be tested in each element test-bench script. Using this command, instead of running (*_INTC_test.lsf) script to run all the element test scripts, CML Compiler only runs the test-bench script that contains the specified keyword(s) and tests the specified FOM. Different FOMs that can be tested with each test-bench script for different elements can be found in elements’ QA description pages. This sub-command is best used with “element_list” sub-command to run certain test-bench scripts for specified elements.
Note also that the above command requires that the compact model library is already installed as a Design Kit within INTERCONNECT. Sometimes, it's necessary to specify a CML file to run QA tests with, if the CML is not installed as a Design Kit within INTERCONNECT. This can be done by using the -k or --cml options like so:
cml-compiler runtests -k path/to/library.cml
or
cml-compiler runtests --cml path/to/library.cml
If there are multiple XML Data Files, and it is necessary to specify which one to use for the element index, this can be done using the -t option as follows:
cml-compiler runtests -t path/to/datafile.xml
If custom QA tests exist, they will be run automatically using the commands above. To run only custom QA tests, use the --custom option:
cml-compiler runtests --custom
More information on custom QA tests can be found in Custom QA tests.
QA test results
When QA tests are run through the terminal, the test results are printed out showing the result of each individual test for every element. The result for each test can have one of three values:
- "PASS" means that the test was completed and the results met specifications.
- "FAIL" means that the test was completed and the result did not meet the specifications. An error message is provided alongside test results with the FAIL status.
- "ERROR" means that the test didn't complete successfully. This can happen for several different reasons, including the absence of INTERCONNECT test files, syntax errors introduced by editing INTERCONNECT test files, or incompatible INTERCONNECT/CML Compiler versions. An error message is provided alongside test results with the ERROR status.
For example, when running the QA tests for the 'wg_strip_straight_c' element within the Lumfoundry Template, the following output is seen:
source.wg_strip_straight_c.wg_strip_straight_c_INTC_test.waveguide_simple_loss_group_delay_qa: PASS
source.wg_strip_straight_c.wg_strip_straight_c_INTC_test.waveguide_simple_loss_group_delay_qa: PASS
source.wg_strip_straight_c.wg_strip_straight_c_INTC_test.waveguide_simple_loss_group_delay_qa: PASS
source.wg_strip_straight_c.wg_strip_straight_c_INTC_test.waveguide_simple_loss_group_delay_qa: PASS
source.wg_strip_straight_c.wg_strip_straight_c_INTC_test.waveguide_simple_neff_ng_qa: PASS
source.wg_strip_straight_c.wg_strip_straight_c_INTC_test.waveguide_simple_neff_ng_qa: PASS
source.wg_strip_straight_c.wg_strip_straight_c_INTC_test.waveguide_simple_neff_ng_qa: PASS
source.wg_strip_straight_c.wg_strip_straight_c_INTC_test.waveguide_simple_neff_ng_qa: PASS
Here, the main test script is wg_strip_straight_c_INTC_test.lsf, which is located within the wg_strip_straight_c subfolder in the /source/ directory. This runs four tests within each of the subtest scripts. This printout is provided in both the terminal as well as the QA_INTC.log file.
Running QA Tests through CLI for Verilog-A CMLs
Using the runtests argument
There are several options that are specific to running QA tests for verilog-A models. To specify the root directory to begin the search for QA test files, use the --directory or -d options:
cml-compiler runtests --veriloga -d testDir/
or
cml-compiler runtests --veriloga --directory testDir/
To specify a filename pattern for the test scripts to search for, use the --pattern option:
cml-compiler runtests --veriloga --pattern "shell_pattern"
Where "shell_pattern" corresponds to a shell-style file pattern to search for. For example, to find and run all tests for models within the directory /source/ that have QA test scripts that start with "wg_strip", run the following command:
cml-compiler runtests --veriloga -d source/ --pattern "wg_strip*"
QA test results
When Verilog-A QA tests are run through the command-line interface, they produce a similar console output to INTERCONNECT QA tests run via the same method. As an example, this is a possible output for the command above:
testGroupIndex (wg_strip_arc_c.wg_strip_arc_c_VA_test.TestWaveguide) ... ok
testEffectiveIndex (wg_strip_arc_c.wg_strip_arc_c_VA_test.TestWaveguide) ... ok
testLoss (wg_strip_arc_c.wg_strip_arc_c_VA_test.TestWaveguide) ... ok
testGroupIndex (wg_strip_sbend_c.wg_strip_sbend_c_VA_test.TestWaveguide) ... ok
testLoss (wg_strip_sbend_c.wg_strip_sbend_c_VA_test.TestWaveguide) ... ok
testEffectiveIndex (wg_strip_sbend_c.wg_strip_sbend_c_VA_test.TestWaveguide) ... ok
testEffectiveIndex (wg_strip_straight_c.wg_strip_straight_c_VA_test.TestWaveguide) ... ok
testLoss (wg_strip_straight_c.wg_strip_straight_c_VA_test.TestWaveguide) ... ok
testGroupIndex (wg_strip_straight_c.wg_strip_straight_c_VA_test.TestWaveguide) ... ok
- "ok" means that the test completed and the results met the specifications of the test.
- "fail" means that the test completed and the results didn't meet the specifications of the test. An error message is provided alongside tests with the "fail" result.
- "error" means that the test didn't complete successfully. This can happen for a number of different reasons, including the absence of Verilog-A test files or netlist files, syntax error introduced by editing Verilog-A test files or netlist lists, out-of-date Spectre, or an inability for Spectre to decrypt Verilog-A models. An error message is provided alongside tests with the "error" result.
Running QA Tests Manually
Sometimes, when developing a CML, it is desirable to obtain more detailed information about the QA tests for a specific element (like how far the test results differ from tolerances). It is possible to manually run the QA tests by navigating to the element data subfolder within /source/, and opening the <element_name>_INTC_test.lsf file within INTERCONNECT for INTERCONNECT CMLs.
If custom QA tests exist, they can be manually run by navigating to the element data subfolder within /source/, and opening the <element_name>_INTC_custom_test.lsf file within INTERCONNECT.
For Verilog-A CMLs, the QA file can be found under QA_veriloga/<element>.scs. These files provide more information about the specifics of certain QA tests and also provide more detailed test results.