How can I trust my testsuite?

Feedback loops

Shercode Holmes

Test coverage

Test coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature >= 1000) {
      return true;
    }
	  
    return false;
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature > 1000) {
      return true;
    }
	
    return false;
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					

Cyclomatic complexity
and
CRAP index

Cyclomatic complexity is a quantitative measure of the number of linearly independent paths through a program's source code

code is hard to understand

code is hard to test properly

code is hard to change

CRAP index

\begin{align} C.R.A.P. &= comp(m)^2 · (1 - \frac{cov(m)}{100})^3 + comp(m) \end{align}

Not all the units of code
are equally important

Churn is a metric representing
how often a unit of code is modified
(ex. number of commits)

https://github.com/bmitch/churn-php

Better, but it's still about coverage,
not the quality of the tests

Avoid mocking frameworks
and testing implementation

James Shore - Testing without mocks

Test doubles
vs
Mocking frameworks
vs
Mocks

class NuclearReactor
{
  public function __construct(
    private TemperatureSensor $sensor
  ) {}

  public function isDangerous(): bool
  {
    if ($this->sensor->currentTemperature() >= 1000) {
      return true;
    }	
		
    return false;
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerousWithMock(): void
  {
    $sensor = $this->createMock(TemperatureSensor::class);
    $sensor->expects($this->once())
      ->method('currentTemperature')
      ->willReturn(2000);

    $nuclearReactor = new NuclearReactor($sensor);
    $this->assertTrue($nuclearReactor->isDangerous());
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerousWithoutMocks(): void
  {
    $sensor = new FakeTemperatureSensor(2000);

    $nuclearReactor = new NuclearReactor($sensor);
    $this->assertTrue($nuclearReactor->isDangerous());
  }
}
					

Cool, but let's find metrics!

Mutation testing

• Create a mutant
• If a test fails -> mutant killed
• If no test fails -> mutant escaped

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature >= 1000) {
      return true;
    }
	
    return false;
  }
}
					

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature > 1000) {
      return true;
    }
 
    return false;
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(1000));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					

How to use it?

To assess the effectiveness of a testsuite!

Setting a minimum MSI in pipeline

While developing a new feature, on a small scope

Ok, I have the tools, now what?

Tools are never the final answer!

Feedback loops are! So...

release frequently

monitor

take educated risks!

learn from failures

Thanks!