How can I trust
my testsuite?

Antonello D'Ippolito

CIAO!

Antonello D'Ippolito

Software engineer

@antodippo

Shercode Holmes

Test coverage

Test coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature >= 1000) {
	  return true;
    } else {
	  return false;
    }
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					
from "Tests Coverage is Dead" by Yotam Kadishay

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature > 1000) {
	  return true;
    } else {
	  return false;
    }
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					
from "Tests Coverage is Dead" by Yotam Kadishay

Cyclomatic complexity
and
CRAP index

PHPUnit dashboard
Cyclomatic complexity is a quantitative measure of the number of linearly independent paths through a program's source code

CRAP index

\begin{align} C.R.A.P. &= comp(m)^2 ยท (1 - \frac{cov(m)}{100})^3 + comp(m) \end{align}

from "Pardon My French, But This Code Is C.R.A.P." by Alberto Savoia
Not all the units of code
are equally important
Churn is a metric representing
how often a unit of code is modified
(ex. number of commits)
from "Getting Empirical about Refactoring" by Michael Feathers

https://github.com/bmitch/churn-php

Better, but it's still about coverage,
not the quality of the tests

Mutation testing

If you make a change that alters the meaning of the code, will one of your tests detect it?

Have you ever used
mutation testing?

Go to menti.com and use the code 6096 4377
OR scan
  • Create a mutant
  • If a test fails -> mutant killed
  • If no test fails -> mutant escaped

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature >= 1000) {
	  return true;
    } else {
	  return false;
    }
  }
}
					

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature > 1000) {
	  return true;
    } else {
	  return false;
    }
  }
}
					

Infection - PHP Mutation Testing Framework
https://infection.github.io/


Live coding




... is risky, so I won't do it


class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					
https://github.com/antodippo/php-testing-playground

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(1000));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					
https://github.com/antodippo/php-testing-playground
https://infection.github.io/guide/mutators.html

PROS

  • Easy to setup and use
  • Could spot critical bugs
  • Improves your testing skills

CONS

  • Slower than other testing tools
  • Can't merge different tools results
  • Some mutants are harmless

How to use it?

To assess the effectivness of a testsuite!

Setting a minimum MSI in pipeline

While developing, on a small scope


						CHANGED_FILES=$(git diff origin/master --diff-filter=AM --name-only | grep src/ | paste -sd "," -);
						INFECTION_FILTER="--filter=${CHANGED_FILES} --ignore-msi-with-no-mutations";
						infection --threads=4 $INFECTION_FILTER
					
https://infection.github.io/guide/how-to.html
Ok, I have the tools, now what?

Tools are never the final answer!

Ship frequently

Monitor

Take educated risks!

Learn from failures

Why automated tests are important?

Go to menti.com and use the code 6074 4568
OR scan

Why?

Reduces cost of change

Enables agility: ship and learn

Short feedback loop

THANKS!


Antonello D'Ippolito | twitter.com/antodippo | antodippo.com Try Infection, Eris and other testing tools here:
https://github.com/antodippo/php-testing-playground