Module eqc_suite

Description
Data Types
Function Index
Function Details

This module provides functions to generate and execute fixed test suites from QuickCheck properties.

Version: 1.42.1

Description

This module provides functions to generate and execute fixed test suites from QuickCheck properties. There are two reasons to do so:

Generated test suites can be run without a QuickCheck licence, so licencees can distribute their properties and generated test suites to their own users, or to other developers, who can then run them in their own context, using a compatible version of QuickCheck Mini.
If a large number of random tests are needed to reach every line of the code under test, then it may be useful to generate a fixed test suite that rapidly covers all the code, and run it as part of regular testing together with the usual random tests. This guarantees high coverage, while retaining the advantages of random testing.

Generated test suites should not be used as a substitute for QuickCheck's usual testing. Their main disadvantages are:

They are not as effective at revealing errors.
Failing tests cannot be shrunk (because the set of tests is fixed).

Example

Suppose a module mut is tested by a property mut_eqc:prop_ok(), and we want to generate a set of test cases for prop_ok() that covers the reachable code. Suppose also we believe that 1000 random tests are enough to cover all the reachable code, with a high probability. Then we can generate the test suite as follows:

cover:compile(mut), so that QuickCheck can measure coverage of this module.
eqc_suite:coverage_based(mut,eqc:numtests(1000,mut_eqc:prop_ok())), to generate the test suite.
eqc_suite:write(mut,v(-1)), to save the generated suite in a file mut.suite.
cover:reset(), to clear the coverage data.
eqc_suite:run(mut_eqc:prop_ok(),mut), to run the tests in mut.suite.
cover:analyse_to_file(mut), to examine the line coverage achieved by the generated tests.

Note that mut.suite and run/2 can be distributed and used without a QuickCheck licence, to repeat the tests later on any platform.

Data Types

file_name()

file_name() = atom() | string()

The name of a file used to store a test suite. Such files have the extension .suite, so for example the atom foo would refer to the file foo.suite.

property()

abstract datatype: property()

A QuickCheck property.

test_case()

test_case() = [any()]

A test case for a QuickCheck property, consisting of a list of values, with one value for each ?FORALL in the property. This is the same kind of value as eqc:counterexample/0 returns.

test_suite()

test_suite() = file_name() | test_suite_representation()

test_suite_representation()

abstract datatype: test_suite_representation()

A structure containing a list of test cases, and optionally additional implementation dependent information.

Function Index

cases/1	Extract the list of test cases from a test suite.
code_coverage/1	Use `eqc_cover` tool to determine code coverage for each test case, and use this as the features.
coverage/3	Use a coverage tool to determine which parts of the code is covered, and use this information as features.
coverage_based/2	Generate a test suite for `Prop` that covers the reachable code in the given modules.
feature_based/1	Given a property, generate a test suite that exercises all the features that are exercised while QuickChecking the property.
line_coverage/2	Use the `cover` tool to determine which lines of code in the specified modules are covered by each test case, and use these lines as features.
random/1	Generate a suite of random tests from a property.
run/2	Equivalent to `run(Prop, Suite, quiet)`.
run/3	Run a previously generated test suite for a property.
split/2	Run a previously generated test suite for a property.
write/2	Save a generated test suite in a file.

Function Details

cases/1

cases(File::test_suite()) -> [test_case()]

Extract the list of test cases from a test suite.

code_coverage/1

code_coverage(Prop::property()) -> property()

Use eqc_cover tool to determine code coverage for each test case, and use this as the features. Of course, modules must be compiled with the eqc_cover parse transform to get any coverage data.

coverage/3

coverage(ResetCoverage::fun(() -> ok), GetCoverage::fun(() -> [any()]), Prop::property()) -> property()

Use a coverage tool to determine which parts of the code is covered, and use this information as features. When using a property constructed by this function the necessary modules must (if necessary) be coverage compiled. The first argument is a function that resets coverage measurement, this function is run before checking each test case. The second argument is a function that provides the coverage information collected. Third argument is the property for which coverage should be measured.

coverage_based/2

coverage_based(Modules::[atom()], Prop::property()) -> test_suite()

Generate a test suite for Prop that covers the reachable code in the given modules. If only a single module is to be covered, then the list of module names may be replaced by an atom. Coverage is determined using the cover tool, and all the modules to be covered must be cover:compiled first. The generated test suite contains a mixture of minimized test cases to reach all the lines of code, and larger more random test cases (which are better at revealing errors).

feature_based/1

feature_based(Prop::property()) -> test_suite()

Given a property, generate a test suite that exercises all the features that are exercised while QuickChecking the property. Features are user-defined, allowing any values to be used as features--for example, lines of code covered, or names of commands executed, might be suitable features to define. Any features can be associated with a test case using eqc:features/2; line coverage features can be defined using line_coverage/2. The generated test suite contains minimised test cases that reach a new feature; test suite generation stops when QuickChecking the property fails to exercise any new features. Thus, if the number of tests specified using eqc:numtests/2 is N, then test suite generation will continue until N consecutive tests fail to exercise any new features.

For example, to generate a minimal test suite that covers my_mod, we could use

  eqc_suite:feature_based(
    eqc_suite:line_coverage(my_mod,
      eqc:numtests(10000,
        my_mod:prop_ok()))).

Here we use line_coverage/2 to use the lines covered in my_mod as features, and eqc:numtests/2 to specify that the property must be tested 10000 times without covering any new lines, for test suite generation to stop. The idea is to choose a large enough number here, that we are unlikely to miss covering any reachable lines of code in the generated test suite. Of course, prop_ok() is the property for which we are generating a test suite.

line_coverage/2

line_coverage(Mod::[atom()], Prop::property()) -> property()

Use the cover tool to determine which lines of code in the specified modules are covered by each test case, and use these lines as features. Of course, the specified modules must be cover:compiled first. If only one module is to be covered, then the first parameter may be an atom rather than a list.

random/1

random(Prop::property()) -> test_suite()

Generate a suite of random tests from a property. The size of the test suite can be controlled by applying eqc:numtests/2 to the property. The test cases are generated in the same way as eqc:quickcheck/1 does, with the exception that duplicate tests are never included in the generated suite.

Generated by EDoc, Sep 18 2017, 16:17:38.