How can I trust
my testsuite?

Antonello D'Ippolito

CIAO!

Antonello D'Ippolito

Software engineer, Scrum master

@antodippo

Shercode Holmes

Test coverage

Test coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature >= 1000) {
	  return true;
    } else {
	  return false;
    }
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					
from "Tests Coverage is Dead" by Yotam Kadishay

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature > 1000) {
	  return true;
    } else {
	  return false;
    }
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					
from "Tests Coverage is Dead" by Yotam Kadishay

Cyclomatic complexity
and
CRAP index

PHPUnit dashboard
Cyclomatic complexity is a quantitative measure of the number of linearly independent paths through a program's source code

CRAP index

\begin{align} C.R.A.P. &= comp(m)^2 ยท (1 - \frac{cov(m)}{100})^3 + comp(m) \end{align}

from "Pardon My French, But This Code Is C.R.A.P." by Alberto Savoia
Not all the units of code
are equally important
Churn is a metric representing
how often a unit of code is modified
(ex. number of commits)
from "Getting Empirical about Refactoring" by Michael Feathers

https://github.com/bmitch/churn-php

Better, but it's still about coverage,
not the quality of the tests

Mutation testing

If you make a change that alters the meaning of the code, will one of your tests detect it?
  • Create a mutant
  • If a test fails -> mutant killed
  • If no test fails -> mutant escaped

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature >= 1000) {
	  return true;
    } else {
	  return false;
    }
  }
}
					

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature > 1000) {
	  return true;
    } else {
	  return false;
    }
  }
}
					

Infection - PHP Mutation Testing Framework
https://infection.github.io/


Live coding




... is risky, so I won't do it


class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					
https://github.com/antodippo/php-testing-playground

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(1000));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					
https://github.com/antodippo/php-testing-playground
https://infection.github.io/guide/mutators.html

PROS

  • Easy to setup and use
  • Could spot critical bugs
  • Improves your testing skills

CONS

  • Slower than other testing tools
  • Can't merge different tools results
  • Some mutants are harmless

How to use it?

While developing, on a small scope

Setting a minimum MSI in pipeline

To assess the effectivness of a testsuite!

Cool!
But how can I improve
the quality of my tests?

Property testing

The idea behind property based testing is to write specifications for input and output and then let the testing framework generate thousands of assertions to verify it.
from How Good Is Your Test Suite? by Luka Muzinic

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(1000));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					

public function testIsDangerousProperties(): void
{
  $nuclearReactor = new NuclearReactor();

  $this->forAll(Generator\choose(0, 999))
    ->then(function (int $temp) use ($nuclearReactor) {
      $this->assertFalse($nuclearReactor->isDangerous($temp));
    });

  $this->forAll(Generator\choose(1000, 99999))
    ->then(function (int $temp) use ($nuclearReactor) {
      $this->assertTrue($nuclearReactor->isDangerous($temp));
    });
}
					
https://github.com/giorgiosironi/eris
Ok, I have the tools, now what?

Tools are never the final answer!

Ship frequently

Monitor

Take educated risks!

Learn from failures

Why?

Reduces cost of change

Enables agility: ship and learn

Short feedback loop

THANKS!


Antonello D'Ippolito | twitter.com/antodippo | antodippo.com Try Infection, Eris and other testing tools here:
https://github.com/antodippo/php-testing-playground