How can I trust
my testsuite?

Antonello D'Ippolito

CIAO!

Antonello D'Ippolito

@antodippo

Shercode Holmes

Test coverage

Test coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature >= 1000) {
	  return true;
    } else {
	  return false;
    }
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature > 1000) {
	  return true;
    } else {
	  return false;
    }
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					

Cyclomatic complexity
and
CRAP index

Cyclomatic complexity is a quantitative measure of the number of linearly independent paths through a program's source code

CRAP index

\begin{align} C.R.A.P. &= comp(m)^2 · (1 - \frac{cov(m)}{100})^3 + comp(m) \end{align}

Not all the units of code
are equally important

Churn is a metric representing
how often a unit of code is modified
(ex. number of commits)

https://github.com/bmitch/churn-php

Better, but it's still about coverage,
not the quality of the tests

Mutation testing

Have you ever used
mutation testing?

• Create a mutant
• If a test fails -> mutant killed
• If no test fails -> mutant escaped

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature >= 1000) {
	  return true;
    } else {
	  return false;
    }
  }
}
					

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature > 1000) {
	  return true;
    } else {
	  return false;
    }
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(1000));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					

How to use it?

To assess the effectivness of a testsuite!

Setting a minimum MSI in pipeline

While developing, on a small scope

						CHANGED_FILES=$(git diff origin/master --diff-filter=AM --name-only | grep src/ | paste -sd "," -);
						INFECTION_FILTER="--filter=${CHANGED_FILES} --ignore-msi-with-no-mutations";
						infection --threads=4 $INFECTION_FILTER
					

Ok, I have the tools, now what?

Tools are never the final answer!

Ship frequently

Monitor

Take educated risks!

Learn from failures

Why automated tests are important?

Why?

Reduces cost of change

Enables agility: ship and learn

Short feedback loop

THANKS!