Batch Equivalence Testing of Multiple Components

This example shows how to create equivalence test cases and test harnesses for multiple components in batch mode using sltest.testmanager.createTestForComponent. Coverage collection and report generation are also included in the example.

The sltestCruiseControl model used in this example has Atomic Subsystem blocks, Virtual Subsystems blocks, and Model Reference Blocks.

Note that this example is configured only for Windows machines.

Open the Model

topModel = "sltestCruiseControl";
load_system(topModel);

Generate the Code

The equivalence tests in this example compare normal mode code and generated code. Embedded Coder is required to generate the code for the model. The model has been configured with appropriate C code generation and coder mapping settings for Windows 64-bit systems. You can change these settings, if desired, before generating the code.

slbuild(topModel);

### Searching for referenced models in model 'sltestCruiseControl'.
### Found 2 model reference targets to update.
### Starting serial model reference code generation build.
### Checking status of model reference code generation target for model 'sltestCruiseControlMode' used in 'sltestCruiseControl'.
### Model reference code generation target (sltestCruiseControlMode.c) for model sltestCruiseControlMode was out of date. sltestCruiseControlMode.c did not exist.
### Starting build procedure for: sltestCruiseControlMode
### Generating code and artifacts to 'Model specific' folder structure
### Generating code into build folder: C:\Users\dschwart\OneDrive - MathWorks\Documents\MATLAB\ExampleManager\dschwart.Bdoc.j2658826\simulinktest-ex89766869\slprj\ert\sltestCruiseControlMode
### Invoking Target Language Compiler on sltestCruiseControlMode.rtw
### Using System Target File: M:\12\dschwart.Bdoc.j2658826\runnable\matlab\rtw\c\ert\ert.tlc
### Loading TLC function libraries
.......
### Initial pass through model to cache user defined code
...
### Caching model source code
............................................................................
### Writing header file sltestCruiseControlMode_types.h
### Writing header file sltestCruiseControlMode.h
### Writing header file rtwtypes.h
### Writing header file sltestCruiseControlMode_private.h
### Writing source file sltestCruiseControlMode.c
.
### TLC code generation complete (took 16.087s).
.### Creating HTML report file index.html
### Saving binary information cache.
### Using toolchain: MinGW64 | gmake (64-bit Windows)
### Creating 'C:\Users\dschwart\OneDrive - MathWorks\Documents\MATLAB\ExampleManager\dschwart.Bdoc.j2658826\simulinktest-ex89766869\slprj\ert\sltestCruiseControlMode\sltestCruiseControlMode.mk' ...
### Successful completion of code generation for: sltestCruiseControlMode
### Checking status of model reference code generation target for model 'sltestDriverSwRequest' used in 'sltestCruiseControl'.
### Model reference code generation target (sltestDriverSwRequest.c) for model sltestDriverSwRequest was out of date. sltestDriverSwRequest.c did not exist.
### Starting build procedure for: sltestDriverSwRequest
### Generating code and artifacts to 'Model specific' folder structure
### Generating code into build folder: C:\Users\dschwart\OneDrive - MathWorks\Documents\MATLAB\ExampleManager\dschwart.Bdoc.j2658826\simulinktest-ex89766869\slprj\ert\sltestDriverSwRequest
### Invoking Target Language Compiler on sltestDriverSwRequest.rtw
### Using System Target File: M:\12\dschwart.Bdoc.j2658826\runnable\matlab\rtw\c\ert\ert.tlc
### Loading TLC function libraries
.......
### Initial pass through model to cache user defined code
..
### Caching model source code
...............................................
### Writing header file sltestDriverSwRequest_types.h
.
### Writing source file sltestDriverSwRequest.c
### Writing header file sltestDriverSwRequest_private.h
### Writing header file sltestDriverSwRequest.h
### TLC code generation complete (took 9.152s).
.### Creating HTML report file index.html
### Saving binary information cache.
### Using toolchain: MinGW64 | gmake (64-bit Windows)
### Creating 'C:\Users\dschwart\OneDrive - MathWorks\Documents\MATLAB\ExampleManager\dschwart.Bdoc.j2658826\simulinktest-ex89766869\slprj\ert\sltestDriverSwRequest\sltestDriverSwRequest.mk' ...
### Successful completion of code generation for: sltestDriverSwRequest
### Simulink cache artifacts for 'sltestCruiseControlMode' were created in 'C:\Users\dschwart\OneDrive - MathWorks\Documents\MATLAB\ExampleManager\dschwart.Bdoc.j2658826\simulinktest-ex89766869\sltestCruiseControlMode.slxc'.
### Simulink cache artifacts for 'sltestDriverSwRequest' were created in 'C:\Users\dschwart\OneDrive - MathWorks\Documents\MATLAB\ExampleManager\dschwart.Bdoc.j2658826\simulinktest-ex89766869\sltestDriverSwRequest.slxc'.
### Starting build procedure for: sltestCruiseControl
### Generating code and artifacts to 'Model specific' folder structure
### Generating code into build folder: C:\Users\dschwart\OneDrive - MathWorks\Documents\MATLAB\ExampleManager\dschwart.Bdoc.j2658826\simulinktest-ex89766869\sltestCruiseControl_ert_rtw
### Invoking Target Language Compiler on sltestCruiseControl.rtw
### Using System Target File: M:\12\dschwart.Bdoc.j2658826\runnable\matlab\rtw\c\ert\ert.tlc
### Loading TLC function libraries
........
### Generating TLC interface API for custom data
.
### Initial pass through model to cache user defined code
.
### Caching model source code
...............................................................................
...............
### Writing header file sltestCruiseControl_types.h
### Writing header file sltestCruiseControl.h
### Writing source file sltestCruiseControl.c
.
### Writing header file sltestCruiseControl_private.h
### Writing source file ert_main.c
### TLC code generation complete (took 23.938s).
### Saving binary information cache.
### Using toolchain: MinGW64 | gmake (64-bit Windows)
### Creating 'C:\Users\dschwart\OneDrive - MathWorks\Documents\MATLAB\ExampleManager\dschwart.Bdoc.j2658826\simulinktest-ex89766869\sltestCruiseControl_ert_rtw\sltestCruiseControl.mk' ...
### Successful completion of code generation for: sltestCruiseControl
### Simulink cache artifacts for 'sltestCruiseControl' were created in 'C:\Users\dschwart\OneDrive - MathWorks\Documents\MATLAB\ExampleManager\dschwart.Bdoc.j2658826\simulinktest-ex89766869\sltestCruiseControl.slxc'.

Build Summary

Model reference code generation targets:

Model                    Build Reason                                       Status           Build Duration
===========================================================================================================
sltestCruiseControlMode  Target (sltestCruiseControlMode.c) did not exist.  Code generated.  0h 2m 1.9212s 
sltestDriverSwRequest    Target (sltestDriverSwRequest.c) did not exist.    Code generated.  0h 0m 36.601s 

Top model targets:

Model                Build Reason                                         Status           Build Duration
=========================================================================================================
sltestCruiseControl  Information cache folder or artifacts were missing.  Code generated.  0h 0m 38.46s  

3 of 3 models built (0 models already up to date)
Build duration: 0h 3m 39.993s

Specify the Components to Test

The example tests all of the atomic subsystems that have nonreusable function packaging in the generated code.

componentsToTest = find_system(topModel,...
    "BlockType","SubSystem",...
    "TreatAsAtomicUnit","on",...
    "RTWSystemCode","Nonreusable function");

To improve traceability of testing artifacts, such as test case and test harness names, customize the default names of the created harnesses. The names will use the component name instead of the owning model name.

sltest.harness.setHarnessCreateDefaults("Name","$Component$_Harness");

Create the Test Cases and Test Harnesses in Batch Mode

Use the sltest.testmanager.createTestForComponent API to create multiple test cases and test harnesses at the same time. Using createTestForComponent, you specify to create a test file and the test filename. You also specify the top model, components to test, and the test type and simulations settings. For the test inputs, you specify to use Simulink Design Verifier to automatically generate the inputs in Microsoft Excel format. If you have other existing inputs, you can add them to the test cases and test both those inputs and the generated inputs.

[tc, status] = ...
    sltest.testmanager.createTestForComponent(...
        "CreateTestFile",true,...
        "TestFile","myB2BTestsDemoEx.mldatx",...
        "TopModel",topModel,...
        "Component",componentsToTest,...
        "TestType","equivalence",...
        "Simulation1Mode","Normal",...
        "Simulation2Mode","Software-in-the-Loop (SIL)",...
        "SLDVTestGeneration","on",...
        "CreateExcelFile",true);

Test Execution

The createTestForComponent API generated the test file, test harnesses, and test input signals in the current working directory. Now, save the harnesses attached to the model, and save the test file.

tf = sltest.testmanager.TestFile("myB2BTestsDemoEx.mldatx");
tf.saveToFile;

Enable coverage collection for test file and run the tests.

cov = getCoverageSettings(tf);
cov.RecordCoverage = true;
cov.MetricSettings = "dcmr";
tf.saveToFile;

resultSet = tf.run;
sltest.testmanager.exportResults(resultSet,"myB2BResults.mldatx");
sltestmgr;

Generate a report with coverage and equivalence test results.

sltest.testmanager.report(...
    resultSet,"myB2BResultsReport.pdf",...
    "IncludeCoverageResult",true,...
    "IncludeSimulationSignalPlots",true,...
    "IncludeComparisonSignalPlots",true,...
    "IncludeTestResults",0,...
    "IncludeSimulationMetadata",true);

Clean Up

bdclose({topModel,'sltestDriverSwRequest','sltestCruiseControlMode'})
sltest.testmanager.clear
sltest.testmanager.clearResults
sltest.testmanager.close