How can I trust
my testsuite?

Antonello D'Ippolito

CIAO!

Antonello D'Ippolito

@antodippo

Shercode Holmes

Test coverage

Test coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature >= 1000) {
	  return true;
    } else {
	  return false;
    }
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature > 1000) {
	  return true;
    } else {
	  return false;
    }
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					

Cyclomatic complexity
and
CRAP index

Cyclomatic complexity is a quantitative measure of the number of linearly independent paths through a program's source code

CRAP index

\begin{align} C.R.A.P. &= comp(m)^2 · (1 - \frac{cov(m)}{100})^3 + comp(m) \end{align}

Not all the units of code
are equally important

Churn is a metric representing
how often a unit of code is modified
(ex. number of commits)

https://github.com/bmitch/churn-php

Better, but it's still about coverage,
not the quality of the tests

Mutation testing

• Create a mutant
• If a test fails -> mutant killed
• If no test fails -> mutant escaped

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature >= 1000) {
	  return true;
    } else {
	  return false;
    }
  }
}
					

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature > 1000) {
	  return true;
    } else {
	  return false;
    }
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(1000));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					

How to use it?

While developing, on a small scope

Setting a minimum MSI in pipeline

To assess the effectivness of a testsuite!

Cool!
But how can I improve
the quality of my tests?

Property testing

The idea behind property based testing is to write specifications for input and output and then let the testing framework generate thousands of assertions to verify it.

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(1000));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					

public function testIsDangerousProperties(): void
{
  $nuclearReactor = new NuclearReactor();

  $this->forAll(Generator\choose(0, 999))
    ->then(function (int $temp) use ($nuclearReactor) {
      $this->assertFalse($nuclearReactor->isDangerous($temp));
    });

  $this->forAll(Generator\choose(1000, 99999))
    ->then(function (int $temp) use ($nuclearReactor) {
      $this->assertTrue($nuclearReactor->isDangerous($temp));
    });
}
					

Ok, I have the tools, now what?

Tools are never the final answer!

Ship frequently

Monitor

Take educated risks!

Learn from failures

Why?

Reduces cost of change

Enables agility: ship and learn

Short feedback loop

THANKS!