Shrinking fallback servers to primary servers

Posted by on Oct 11, 2015 in Blog

Shrinking fallback servers to primary servers Imagine you are running a large system with some primary and some fallback servers. You have at least one, up to three primary servers and zero to four fallback servers. You run tests on a random configuration to make sure your system behaves as expected. One of those tests fails and you want to shrink it to a minimal counter example. A minimal counter example often takes the configuration into account. Years ago we showed how to test elevator control software using QuickCheck. The property first selected a random number of floors and elevators and then run a sequence of random commands. As soon as such sequence failed, it would shrink the configuration to the minimal number of floors and elevators needed to provoke this error. The idea is that analyzing a failure in a small configuration or a default (i.e., well known) configuration is easier than analyzing it in a large, tricky configuration. Moreover, if we cannot shrink the configuration to for example 2 floors, then we know that we need at least 3 floors for the error to happen, giving us additional information about the possible causes. At Basho they test their software against many random configurations of primary and fallback...

Learn More

How to test inner loops

Posted by on Nov 17, 2014 in Blog

How to test inner loops In this blog post we will look at something that has kept me busy for a couple of days, writing tests for code with inner loops. Loops are very common and although it is often easy to understand how they are intended to work, it can be surprisingly difficult to test them. Fortunately, QuickCheck has a very cool block/unblock feature that allows you to split a loop into smaller, more manageable chunks that can be tested in a straightforward way. To illustrate how the block and unblock feature works I will implement a simple TCP/IP time server that lets users connect to the port it is listening to and then replies with a text message telling what time it is. It is simple but complicated enough for this blog post. A first (loop-less) version of the time server Let us start with a very simple version of the time server without loops that only allows for a single user to connect to it. This will serve as an introduction to how to write QuickCheck component tests and will set the stage for the next section where we add an inner loop. If you have already written QuickCheck tests, the material in this section will...

Learn More

Generating Fixed Test Suites with QuickCheck

Posted by on Oct 16, 2014 in Blog

Generating Fixed Test Suites with QuickCheck The great strength of QuickCheck is that we never run out of tests to run: QuickCheck can always find more and more combinations of things to test, and so rare bugs that depend on several features interacting can eventually be found. Nevertheless, sometimes you just want to test fast, and be sure that you did indeed test everything at least once. QuickCheck is able to generate and run saved test suites with a variety of properties, which can be used as a fast sanity check in between longer test runs using random generation of test cases. In this blog post, we’ll show how to use recent additions to QuickCheck to generate better test suites. The example we’re going to use is a simple specification of the Erlang process registry, which does both positive and negative testing, and moreover kills processes randomly to test the registry’s behaviour when processes crash. You can find the source code we’re starting from here. It’s a state machine specification, with a property instrumented to collect the proportions of each command in the generated tests. prop_registry() -> ?FORALL(Cmds, commands(?MODULE), begin [catch unregister(N) || N <- ?names], {H, S, Res} = run_commands(?MODULE,Cmds), pretty_commands(?MODULE, Cmds, {H, S, Res}, aggregate(command_names(Cmds), Res ==...

Learn More

Adding a GitHub repository to QuickCheck-CI

Posted by on Oct 10, 2014 in Blog

Adding a GitHub repository to QuickCheck-CI In this blog article we are going to explain how you can add a repository on GitHub to QuickCheck Continuous Integration. In contrast to other continuous integration systems our CI system is specialised in property based testing. After the code has been built, we run QuickCheck on the properties that are present in the repository. Using these properties we can thoroughly test the software and provide detailed feedback in case a property fails. In addition we generate coverage information, which can be used to spot dead code or to find the location of a bug. Unfortunately many projects on GitHub haven’t defined properties yet. Many projects, however, do have unit tests. A property abstracts away from a single unit tests, and covers many unit tests. We are going to show how to define a property based on a unit test. Apache Couchdb We are going to use Apache CouchDB (with git revision 6fc1124) as an example project. Apache CouchDB, commonly referred to as CouchDB, is an open source database that focuses on ease of use. It is a NoSQL database that uses JSON to store data, JavaScript as its query language using MapReduce, and HTTP for an API. The project consists of many Erlang modules and contains two test suites: one based on ETAP and...

Learn More

Checksum property for AUTOSAR

Posted by on Jan 21, 2013 in Blog

Checksum property for AUTOSAR Property based testing is based upon the idea that one writes a property about the software under tests and from that property the tool QuickCheck automatically generates test cases to test your software. Properties can be difficult to formulate, since a lot of software has grown and the once clear ideas have been polluted and clear properties have been obfuscated. That is, if the software has not been developed using property based development. However, there is a simple property that comes in handy very often: one version of the software should for a certain input domain be the same as a different version. This comparison with other versions or even other implementations can often reveal errors introduced by optimizations or compatibility issues between competing implementations. In this post we take a simple example of a table based implementation and a computation based computation of a checksum algorithm. The example was motivated by the implementation of the CRC library in the AUTOSAR standard. Two differently configured C programs should give equal results on the same input. This is a commonly occurring phenomena, but the solution presented is kept simple and does not take stateful systems into account; this extension is possible and subject for a later post. Modelling CRC routines Computing...

Learn More