Is it acceptable to test based on test output data rather than input data in unit tests?

Question

I'm used to write unit tests with assertions which are based on the input, e.g. (hopefully self-explanatory and let's assume that using random test data is fine)

int a = random();
int b = random();
Adder instance = new Adder();
int expResult = a+b;
int result instance.add(a, b);
assertEquals(expResult, result);

Let's assume that Adder.add has a lot of side effects which I cannot imagine, so that this test would make sense.

Now, I encountered a situation where it would make sense to create assertions based on the output, e.g.

int a = random();
int b = random();
Multiplier instance = new Multiplier();
int result = instance.multiply(a, b);
if(isPrimeNumber(result)) {
    assertTrue(a == 1 || b == 1);
}else {
    //some other assertions...
}

Yes, this is a non-sense test and it tests more the functioning of rational numbers than anything else, but it illustrates the difference between basing assertions on input exclusively and making the output/test result influence assertions.

I'm assuming that I cover all possible distinct output states of the test - just like I'd assume that I'm covering all possible input states.

Answer 1

In this example, there is a certain risk of having instance.multiply(a, b) producing a prime number where it should not, (maybe for a pair 1,10 it delivers 11) and then leaving out the "other assertions" section.

More general, the fact you need some condition to test on the output data is a sign that your test data generation does not behave deterministic, so you cannot easily pick the right choice of assertions beforehand. That is the main problem I see here, since non-deterministic tests have the very nasty property of not creating reproducable outcomes.

Let us take your second example: if there would be a fixed set of deterministic test data (even if it was produced with a random generator once), for each pair (a,b) of numbers it would be determinable beforehand if their product should be a prime number or not. A better test then could look like this:

 [TestCase(1,41,true)]
 [TestCase(2,3,false)]
 [TestCase(10,1,false)]
 void TestMultiplier(int a, int b, bool prodIsPrime)
 {
     Multiplier instance = new Multiplier();
     int result = instance.multiply(a, b);
     assertEqual(prodIsPrime, isPrimeNumber(result));
     if(!prodIsPrime)
     {
       // some other tests
     }
  }

Of course, you might then refactor this further and split this into two tests, one for prime products and one for non-primes, as @Fabio wrote.

Answer 2

Asserting certain relations between the input and output data without specifying concrete test values is a valid testing strategy, also known as property testing. This has the advantage that a large input space can be covered easily with very little code. Property testing moves away from testing some specific hand-chosen examples, towards testing laws that must hold for the whole input space.

But this only works under the following conditions:

• You have a sensible mechanism to generate input values. Choosing values at random can be part of that strategy, but is insufficient. A mixture between random values and special interesting values is best (e.g. boundaries of your input domain). When using random values, you must record the seed in order to make your tests reproducible.

• The system under test is pure, so that it can be run repeatedly, in arbitrary order, and quickly.

• The system under test runs reasonably quickly, so that many instances can be exercised.

• The properties must be very cheap to check. E.g. a primality test would be inconveniently expensive.

• The properties must not simply rephrase the implementation being tested, or they are worthless. E.g. in your example, a property that checks result == a + b would fail to catch problems relating to numeric overflow. A more interesting property would be if (a > 0 && b > 0) assert(result > a && result > b).

• Properties don't necessarily have to restrict themselves for a single input. E.g. we might also assert the commutativity property add(a, b) == add(b, a). It is still sensible to write separate tests for properties around specific input values, e.g. add(a, 0) == a.

Property-based testing was largely popularized by the Haskell QuickCheck library. Now, comparable frameworks exist in a large variety of languages. Their main value is that they assist in generating interesting input values. They may also assist in exercising only a subset of possible input combinations, in order to avoid exponential explosion of test cases.

So using a hand-rolled property testing approach, we might write the test like this (Python-like pseudocode):

def generate_integers(rng: Random) -> Iterator[int]:
    # interesting values around zero and 1
    yield from [-1, 0, 1, 2, 10, 11]
    # interesting boundary values
    yield from [INT_MIN, INT_MIN + 1, INT_MAX - 1, INT_MAX]
    # extra random values
    5 times:
      yield rng.next_int()

def test_commutativity(rng):
    a_values = list(generate_integers(rng))
    b_values = list(generate_integers(rng))
    for a in a_values:
        for b in b_values:
            assert add(a, b) == add(b, a)

def test_identity(rng):
    for a in generate_integers(rng):
        assert add(a, 0) == a

def test_inverse(rng):
    a_values = list(generate_integers(rng))
    b_values = list(generate_integers(rng))
    for a in a_values:
        for b in b_values:
            result = add(a, b)
            assert result - a == b
            assert result - b == a