SuperTest

SuperTest Compiler Test and Validation Suite for C and C++

SuperTest is the test and validation suite for C and C++ compilers and libraries that has tracked the (ISO) language specifications for more than 40 years. It is a complete validation environment with:

• A huge collection of hand-written test
• A large body of generated tests
• A configurable test driver to easily set up and run validations
• A report generator that provides comprehensive overviews of validation results in multiple formats, including HTML and JUnit XML, and will immediately identify regressions of the latest compiler and library version compared to a designated reference
• SuperTest comes with a bundle of ~24K hand-written files generating Millions lines of code to achieve all standard testing.

SuperTest includes many more unique features that are not found in other test suites:

• Very easy and quick installation and configuration
• Can be used to verify deeply embedded freestanding C implementations on memory constraint devices (down to 4K memory)
• Clear and easy reporting in text, HTML, JUnit XML and CSV
• The CGtrainer (Code Generator trainer), a suite of nearly one thousand tests aimed at bootstrapping new compiler development
• Powerful testing of the code generator/back-end
• The Depth Suite is a collection of generated tests that exhaustively verifies arithmetic correctness relative to the target machine’s data model. Depth Suites for many data models are included, but if your data model is not there, we will create one for you free of charge
• Parallel testing automatically splits validation runs into a preconfigured numbers of threads to speed up the validation
• Testing of libraries
• A production rule driven test generator to create pseudo-random variations of a particular test
• Easy addition of your own specific tests
• Excellent professional support
• Multi-processing to execute tests faster
• Supports bare metal targets
• Traceability to the sections of the ISO standards
• Runs on Windows and Linux
• Flexible test generators
• Full control over the test sets
• Python Configuration plugins
• Reporting tools: SuperTest generates high quality reports that are easy to read and analyze

ISO Standard Conformance Testing

SuperTest can be used to qualify compilers and libraries for functional safety based on standards like ISO 26262, IEC 61508, EN 50128, IEC 62304. Even if you are a compiler end-user and can’t fix the C/C++ compiler, SuperTest will help you know what C/C++ syntax to avoid so you don’t have a runtime issue.

The SuperTest suite covers the following ISO specifications:

ISO/IEC 9899:1990 C90 standard
ISO/IEC 9899:1999 C99 standard
ISO/IEC 9899:2011 C11 standard
ISO/IEC 9899:2018 C18 standard
ISO/IEC TR 18037:2008 Embedded C extension
ISO/IEC 14882:2003 C++03
ISO/IEC 14882:2011 C++11
ISO/IEC 14882:2014 C++14
ISO/IEC 14882:2017 C++17
ISO/IEC 14882:2020 C++20

The test suite is systematically structured so that it is clear what section of which standard is tested by each test. Therefore, it is also easy to configure which language version should be tested when executing the test suite.

Additional SuperTest Features

• Optimization correctness
• C and C++ Standard Library Testing
• Code Generator development
• Data Model specific arithmetic
• Calling convention testing
• In-depth arithmetic testing

SuperTest Can Qualify Any C or C++ Compiler

SuperTest can be configured to qualify any C or C++ compiler toolchain for any execution environment. We provide example configurations for many of the compiler toolchains and targets listed below. Using an example configuration, getting started with using SuperTest for the qualification of the compiler toolchain and target that you are using is easy. SuperTest targets open source based compilers such as GCC (all versions), LLVM (Clang, all versions), STM32CubeIDE, TI Code Composer, Renesas 22 Studio; and most popular proprietary C/C++ compilers in the market. Email info@emprog.com to check if you compiler has a ready-to-use SuperTest for it.

SuperTest Testimonials

“With SuperTest, we can now reliably tell our customers that our compiler verification covers the entire C and C++ language standard. SuperTest includes tests for the optimizations that we often develop for our customers in order to meet their performance or code size requirements. As a result, we can run tests that target every part of the compiler,” Customer’s testimony after trying to develop their own test suit based on the LLVM and GCC Torture test suites.

“Before, when customers asked us how do we really know our compilers work, we didn’t always have a very good answer. After using SuperTest, now we can say with a very high level of confidence that they do.” Customer’s testimony

Functional Safety

Functional safety is one of the most important aspects of many embedded systems today, especially in sectors such as automotive, railway, industrial automation and medical. SuperTest and SuperGuard can be used to qualify compilers and libraries for functional safety.

Compiler qualification is the process that can be used to develop the required confidence in the compiler. Compiler testing against the language specification is the proper method for this. SuperTest, with its traceability from the C and C++ language standards to the test, is the best test suite for compiler qualification.

Library qualification is even more critical because the library contains code that is linked into the application and loaded onto the target device. If a library component is defective, the functional safety of the entire application is therefore jeopardized. SuperGuard can be used to support qualification of standard library implementations for safety critical applications.

There are different functional safety standards for which you can use SuperTest and SuperGuard to qualify your compiler and library.

ISO 26262
ISO 26262 is the functional safety standard for electrical and electronic systems in series production passenger cars and other automotive categories. It requires qualification of all (electrical and electronic) components with respect to safety. The qualification process is described in the standard.

IEC 61508
IEC 61508 is the international umbrella standard for functional safety. This standard and other standards derived from it are used in all kinds of industries with requirements on reliability and safety, for example process industries, nuclear power plants and railway control systems.

EN 50128/EN 50716
EN 50128/EN 50716 are European standard for safety related software in railway applications

IEC 62304
IEC 62304 is the international standard which specifies life cycle requirements for the development of medical software and medical devices.