Combine can generate you test data that satisfies a T-wise (also known as t-way) Coverage. This means that the test set contains all T-combinations of parameter values (or block of values) of an SUT (System Under Test). It can also generate valid block values into the test set. Short example:
SUT = [2, 2, 2] meaning that SUT has 3 input parameters, each of which has possible 2 values (or 2 classes/blocks of values to be precise). Let's name the parameters A, B, and C.
T = 2 (this means we want to cover all possible tuples of values, i.e. pair-wise coverage)
tuples to cover:
| A | B | C |
|---|---|---|
| 1 | 1 | |
| 1 | 2 | |
| 2 | 1 | |
| 2 | 2 | |
| 1 | 1 | |
| 1 | 2 | |
| 2 | 1 | |
| 2 | 2 |
All the tuples above can be covered by only 4 tests:
| A | B | C |
|---|---|---|
| 1 | 1 | 1 |
| 1 | 2 | 2 |
| 2 | 1 | 2 |
| 2 | 2 | 1 |
All the user need is to specify the options for test data generation:
and then the SUT parameters. You can choose from the following 10 basic data types:
These are to be specified with the following options:
When a parameter is added, you can define values for each parameter block. To do that, you will need to write it in a language that allows you to specify block properties of given parameter. Let's look at the properties and languages in the next section.
You can also define a constraint that forbids certain block combination. See this section for more.
| data type (example identifier used in constraints) | things you want to define | what to write into 'block value constraint' field |
|---|---|---|
| integer (a) | block string description | "negative values" |
| number value restriction with operators <, <=, >, >=, =, != | a > 42 | |
| use and operator to create intervals | a > 42 and a < 100 and a != 84 | |
| create split interval with or operator, don't forget parentheses (for example: a is negative or in interval <42, 100)) | a < 0 or (a >= 42 and a < 100) | |
| float (b) | block string description | "non zero values" |
| number value restriction with operators <, <=, >, >=, =, != | b < 42.42 | |
| use and operator to create intervals | b > 0.12 and b < 0.14 and b != 0.13 | |
| create split interval with or operator, don't forget parentheses (for example: b is positive or in interval <-13.25, -5.5)) | b > 0.0 or (b >= -13.25 and b < -5.5) | |
| define value as not a number | b = nan | |
| define value as infinity | b = inf | |
| define value as -infinity | b = -inf | |
| boolean | you cannot specify anything, it specifies itself :). Block number 1 means true, block number 2 means false. | |
| enum | one value in enum | you can write whatever you want |
| string (s) | block string description | "two-character strings" |
| define block value as one of the items in list, use oneof() operator | oneof(Katy, Jane, Lucy, Marge) | |
| choose value to be equal (=) or not equal (!=) to some string | s = "Hello World!" | |
| length (len) of the string, you can define it the same way as integer (operators, and, or), (bug: maximal minimal-length is 10: len>9, minimal maximal-length is 20: len<=20)don't forget parentheses | (len > 12 and len <=20) or (len > 3 and len < 8) | |
| choose category, you have three options: alphanum, alphabetic, num | category = alphabetic | |
| mix following options together with and | category = alphanum and len > 5 and len <= 25 and s != "123456" | |
| date (d) | block string description | "december 2012" |
| date value restriction with operators <, <=, >, >=, =, != and YYYY-MM-DD format | d >= 2001-03-12 | |
| use and operator to create intervals | d > 2008-07-01 and d < 2008-08-31 | |
| create split interval with or operator, don't forget parentheses (for example: d is date in 21th century or in interval from 1990-02-01 to 1992-06-24) | d >= 2000-01-01 or (d >= 1990-02-01 and d <= 1992-06-24) | |
| time (t) | block string description | "afternoon" |
| time value restriction with operators <, <=, >, >=, =, != and HH:MM:SS format | t >= 8:00:00 | |
| use and operator to create intervals | t > 7:59:59 and t <= 13:40:00 | |
| create split interval with or operator, don't forget parentheses (for example: t is before noon or in interval from 5pm to 8pm) | t < 12:00:00 or (t >= 17:00:00 and t <= 20:00:00) | |
| block string description | "any example.com address" | |
| set local part (local_part) of the email as equal (=) or not equal (!=) to some string | local_part != "headmaster" | |
| set domain part (domain) of the email as equal (=) or not equal (!=) to some string | domain = "example.com" | |
| set length of the whole email (len), local part (local_part_len) or domain (domain_len) using operators <, <=, >, >=, =, != | len < 40 and local_part_len >= 6 | |
| mix it up with and operator | domain_len > 6 and local_part != "admin" and local_part_len <= 25 and domain != "example.com" | |
| telephone number | block string description | "any 10 digit number" |
| set prefix (prefix) in the telephone number as equal (=) or not equal (!=) to some string | prefix = "+420" | |
| set the number part (number) in the telephone number as equal (=) or not equal (!=) to some string | number = "987 654 321" | |
| define the length of the number part (number_len) with operators <, <=, >, >=, =, != | number_len > 8 and number_len < 12 | |
| mix it up with and operator | prefix != "+420" and number_len = 9 and number != "123456789" | |
| ID string (i) | block string description | "product key to some software" |
| choose value to be equal (=) or not equal (!=) to some string | i = "182KI-IEUHA-827YH" | |
| define delimiter bewtween blocks | delimiter = "-" | |
| define number of blocks (block_count) in ID string (with = operator) | block_count = 4 | |
| define length of blocks (block_len) in ID string (with = operator) | block_len = 6 | |
| mix it up with and operator | block_len = 3 and block_count = 4 and i != "NMK-21d-1f2-817" and delimiter = '-' |
The table shows some ways to specify blocks in each parameter data type, here is some more information about the data types:
It is possible to restrict some combinations of parameter blocks, that don't make any sense. For example consider following situation:
We want to test out a web app, we choose to try different browsers, operation systems and internet connections, so we get 3 parameters with following blocks:
In this situation it doesn't make sense to test Microsoft Edge on other OS than Windows, the same situation emerges with Safari. So we want to specify constraints that tell us which combinations are not valid. Following demostrations of constraint language is shown on 3 parameters with IDs browser, os, connection and blocks are in the same order as in the example:
| basic language structures | usage example |
|---|---|
| use id.number_of_block to identify 1 specific block (for example Edge as 5th block in browser parameter) | browser.5 |
| use logical operators and, or, xor, -> (implication),
<-> (equivalence) and ! (not) (don't forget parentheses) | browser.5 -> (!os.2 and !os.3) |
A characteristic is something that divides the input into blocks (in which all variables has equal functionality for testing). Here are some common situations that can help you with your testing (and there is also an example how to write them into Combine):
In Combine you use integer of float data types.
Suitable for situations, when we have parameter with interval of valid values (other values outside the interval are invalid). So we have 2 blocks, one valid, one invalid. Into block value constraint you put:
for interval of valid values (-inf, x> where a as parameter identificator and x is some integer, for example 42 (you know how to specify intervals from here).
When testing math operations you need to test these blocks (written in block definitions):
for identifier a, e being smallest possible difference between 2 numbers, for integers is e = 1, when working with floats, tester needs to decide about the accuracy that is needed. NaNis used only when working with floats, MAX and MIN represent maximal and minimal possible value.
Sometimes code implies some boundary values, that we want to wanna test around to avoid error by one. Than we have 3 blocks for each boundary, boundary - e, boundary and boundary + e:
for boundary = 42 and e = 1 (because we work with ints). So 1 boundary -> 3 blocks, 2 boundaries -> 6 blocks etc.
Similar to numbers, we have some boundary date, epoch. 1 epoch means 3 blocks, so we have the same situations as with boundary values (e = 1 day, here a is id and epoch = 1970-01-01):
Used for example for being late to school etc., similar to valid intervals at numbers, let's take school starting at 8am as an example (a is id)
When testing string length (for example form input of MAX number of characters), we have to consider empty string, string of length < MAX, but not empty, string length = MAX length and string len = MAX length + 1 (example max = 42):
This example shows how to use oneof() operator in string data type, consider situation when you want to test input name matching with sex, then name will be divided into 2 blocks, male and female names:
parameter name:
parameter sex:
When testing the ID string (key), we can divide input domain input blocks based on the string format, that means number of block in ID string and length of each block and the delimiter (1 char between blocks in ID string). That we have 4 combinations, 1. with correct length and correct delimiter, 2. with correct length but incorrect delimiter, 3. with incorrect length but correct delimiter and 4. with incorrect length and incorrect delimiter:
Phone numbers can be divided by the internation prefix to internation numbers and number from your country, For example if you live in the Czech Republic (prefix +420), it would look like this:
Email can be sorted based on it's domain, for example if your company's domain was exmaple.com, then we would have 2 blocks (again can be mixed up with other properties for which you might need more than 2 blocks):
Configurations of an SUT is great situations to use combinatorial testing, you can represent 2-state options as booleans and N-state options as enums.
Can be represented as enum to represent 3 blocks:
Container is an object with variable number of items, for example buffer, C string etc. This can be represented by enum data type, enum values should cover these blocks: