How can I trust my testsuite?

Antonello D'Ippolito - Amsterdam PHP, April 2024

Feedback loops


🇮🇹 CIAO! 🤌

Antonello D'Ippolito

Software engineer @ Mollie

@antodippo

Shercode Holmes

Test coverage

Test coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature >= 1000) {
      return true;
    }
	  
    return false;
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					
from "Tests Coverage is Dead" by Yotam Kadishay

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature > 1000) {
      return true;
    }
	
    return false;
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					
from "Tests Coverage is Dead" by Yotam Kadishay
A lot of the value of code coverage data is to highlight not what’s covered, but what’s not covered.

from "Code Coverage Best Practices", Google Testing Blog

Cyclomatic complexity
and
CRAP index

Cyclomatic complexity is a quantitative measure of the number of linearly independent paths through a program's source code
High cyclomatic complexity means:

code is hard to understand

code is hard to test properly

code is hard to change

CRAP index

\begin{align} C.R.A.P. &= comp(m)^2 · (1 - \frac{cov(m)}{100})^3 + comp(m) \end{align}

from "Pardon My French, But This Code Is C.R.A.P." by Alberto Savoia
Not all the units of code
are equally important
Churn is a metric representing
how often a unit of code is modified
(ex. number of commits)
from "Getting Empirical about Refactoring" by Michael Feathers

https://github.com/bmitch/churn-php

Better, but it's still about coverage,
not the quality of the tests

Avoid mocking frameworks
and testing implementation

James Shore - Testing without mocks

Test doubles
vs
Mocking frameworks
vs
Mocks


class NuclearReactor
{
  public function __construct(
    private TemperatureSensor $sensor
  ) {}

  public function isDangerous(): bool
  {
    if ($this->sensor->currentTemperature() >= 1000) {
      return true;
    }	
		
    return false;
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerousWithMock(): void
  {
    $sensor = $this->createMock(TemperatureSensor::class);
    $sensor->expects($this->once())
      ->method('currentTemperature')
      ->willReturn(2000);

    $nuclearReactor = new NuclearReactor($sensor);
    $this->assertTrue($nuclearReactor->isDangerous());
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerousWithoutMocks(): void
  {
    $sensor = new FakeTemperatureSensor(2000);

    $nuclearReactor = new NuclearReactor($sensor);
    $this->assertTrue($nuclearReactor->isDangerous());
  }
}
					
Cool, but let's find metrics!

Mutation testing

If you make a change that alters the meaning of your code, will one of your tests detect it?
  • Create a mutant
  • If a test fails -> mutant killed
  • If no test fails -> mutant escaped

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature >= 1000) {
      return true;
    }
	
    return false;
  }
}
					

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature > 1000) {
      return true;
    }
 
    return false;
  }
}
					

Live coding




... is risky, so I won't do it


class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					
https://github.com/antodippo/php-testing-playground

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(1000));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					
https://github.com/antodippo/php-testing-playground

PROS

  • Easy to setup and run
  • Might spot potential critical bugs
  • Improves your testing skills

CONS

  • Can't merge different tools results
  • Some mutants are harmless
  • Slower than other testing tools

How to use it?

To assess the effectiveness of a testsuite!

Setting a minimum MSI in pipeline

While developing a new feature, on a small scope

Ok, I have the tools, now what?

Tools are never the final answer!

Feedback loops are! So...

release frequently

monitor

take educated risks!

learn from failures

Thanks!

Twitter: @antodippo | Mastodon: @antodippo | antodippo.com