Code Smell 52 - Fragile Tests

Tests are our safety nets. If we don't trust on their integrity, we will be in great danger

2 March, 2023

Contributors

Maxi Contieri

@mcsee

Tests are our safety nets. If we don't trust their integrity, we will be in great danger

TL;DR: Don't write non-deterministic tests.

Problems

Determinism
Confidence loss
Wasted time

Solutions

The tests should be in complete control. There should be no space for erratic behavior and degrees of freedom.
Remove all tests coupling.

Coupling - The one and only software design problem

Examples

Fragile, Intermittent, Sporadic, or Erratic tests are standard in many organizations.

Nevertheless, they mine the developer's trust.

We must avoid them.

Sample Code

Wrong

import static org.junit.Assert.assertEquals;

import org.junit.Test;

import components.set.Set;
import components.set.Set1L;

public abstract class SetTest {

    protected abstract Set<String> constructor();

    @Test
    public final void testAddEmpty() {
        Set<String> s = this.constructor();
        s.add("green");
        s.add("blue");
        assertEquals("{green. blue}", s.toString());
       // This is fragile since it depends on set sort (which is not defined)
    }   
}

Right

import static org.junit.Assert.assertEquals;

import org.junit.Test;

import components.set.Set;
import components.set.Set1L;

public abstract class SetTest {

    protected abstract Set<String> constructor();

    @Test
    public final void testAddEmpty() {
        Set<String> s = this.constructor();
        s.add("green");
        assertEquals("{green}", s.toString());
    }   

    @Test
    public final void testEntryAtSingleEntry() {
        Set<String> s = this.createFromArgs("red");
        Boolean x = s.contains("red");
        assertEquals(true, x);
    } 
}

Detection

Detection can be done with test run statistics.

It is very hard to put some tests in maintenance since we are removing a safety net.

More Info

https://softwareengineering.stackexchange.com/questions/109703/how-to-avoid-fragile-unit-tests

Relations

Code Smell 76 - Generic Assertions

Conclusion

Fragile tests show system coupling and not deterministic or erratic behavior.

Developers spend lots of time and effort fighting against these false positives.

Credits

Photo by Jilbert Ebrahimi on Unsplash

The amateur software engineer is always in search of magic.

Grady Booch

Software Engineering Great Quotes

This article is part of the CodeSmell Series.

How to Find the Stinky Parts of your Code