How can I trust my testsuite?

Antonello D'Ippolito
PUG Milano, 22/2/23

CIAO!

Antonello D'Ippolito

Software engineer

@antodippo

Let's make it interactive!

Shercode Holmes

Test coverage

Test coverage is a measure used to describe the degree to which the source code of a program is executed when a particular test suite runs

Poll time!


class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature >= 1000) {
      return true;
    }
	  
    return false;
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					
from "Tests Coverage is Dead" by Yotam Kadishay

class NuclearReactor
{
  public function isDangerous(int $temperature): bool
  {
    if ($temperature > 1000) {
      return true;
    }
	
    return false;
  }
}
					

class NuclearReactorTest extends TestCase
{
  public function testIsDangerous(): void
  {
    $nuclearReactor = new NuclearReactor();
    $this->assertFalse($nuclearReactor->isDangerous(500));
    $this->assertTrue($nuclearReactor->isDangerous(2000));
  }
}
					
from "Tests Coverage is Dead" by Yotam Kadishay
A lot of the value of code coverage data is to highlight not what’s covered, but what’s not covered.

from "Code Coverage Best Practices", Google Testing Blog

Cyclomatic complexity
and
CRAP index

PHPUnit dashboard
Cyclomatic complexity is a quantitative measure of the number of linearly independent paths through a program's source code
High cyclomatic complexity means:
  • code is hard to understand
  • code is hard to test properly
  • code is hard to change
  • CRAP index

    \begin{align} C.R.A.P. &= comp(m)^2 · (1 - \frac{cov(m)}{100})^3 + comp(m) \end{align}

    from "Pardon My French, But This Code Is C.R.A.P." by Alberto Savoia
    Not all the units of code
    are equally important
    Churn is a metric representing
    how often a unit of code is modified
    (ex. number of commits)
    from "Getting Empirical about Refactoring" by Michael Feathers

    https://github.com/bmitch/churn-php

    Poll time!

    Better, but it's still about coverage,
    not the quality of the tests

    Avoid mocking frameworks
    and testing implementation

    James Shore - Testing without mocks

    Mocks
    vs
    Mocking frameworks
    vs
    Test doubles

    
    class NuclearReactor
    {
      public function __construct(
        private TemperatureSensor $sensor
      ) {}
    
      public function isDangerous(): bool
      {
        if ($this->sensor->currentTemperature() >= 1000) {
          return true;
        }	
    		
        return false;
      }
    }
    					
    
    class NuclearReactorTest extends TestCase
    {
      public function testIsDangerousWithMock(): void
      {
        $sensor = $this->createMock(TemperatureSensor::class);
        $sensor->expects($this->once())
          ->method('currentTemperature')
          ->willReturn(2000);
    
        $nuclearReactor = new NuclearReactor($sensor);
        $this->assertTrue($nuclearReactor->isDangerous());
      }
    }
    					
    
    class NuclearReactorTest extends TestCase
    {
      public function testIsDangerousWithoutMocks(): void
      {
        $sensor = new FakeTemperatureSensor(2000);
    
        $nuclearReactor = new NuclearReactor($sensor);
        $this->assertTrue($nuclearReactor->isDangerous());
      }
    }
    					

    Mutation testing

    If you make a change that alters the meaning of your code, will one of your tests detect it?

    Poll time!

    • Create a mutant
    • If a test fails -> mutant killed
    • If no test fails -> mutant escaped
    
    class NuclearReactor
    {
      public function isDangerous(int $temperature): bool
      {
        if ($temperature >= 1000) {
          return true;
        }
    	
        return false;
      }
    }
    					
    
    class NuclearReactor
    {
      public function isDangerous(int $temperature): bool
      {
        if ($temperature > 1000) {
          return true;
        }
     
        return false;
      }
    }
    					

    Infection - PHP Mutation Testing Framework
    https://infection.github.io/



    Live coding




    ... is risky, so I won't do it

    
    class NuclearReactorTest extends TestCase
    {
      public function testIsDangerous(): void
      {
        $nuclearReactor = new NuclearReactor();
        $this->assertFalse($nuclearReactor->isDangerous(500));
        $this->assertTrue($nuclearReactor->isDangerous(2000));
      }
    }
    					
    https://github.com/antodippo/php-testing-playground
    
    class NuclearReactorTest extends TestCase
    {
      public function testIsDangerous(): void
      {
        $nuclearReactor = new NuclearReactor();
        $this->assertFalse($nuclearReactor->isDangerous(500));
        $this->assertTrue($nuclearReactor->isDangerous(1000));
        $this->assertTrue($nuclearReactor->isDangerous(2000));
      }
    }
    					
    https://github.com/antodippo/php-testing-playground

    PROS

    • Easy to setup and use
    • Might spot critical bugs
    • Improves your testing skills

    CONS

    • Can't merge different tools results
    • Some mutants are harmless
    • Slower than other testing tools

    How to use it?

    To assess the effectiveness of a testsuite!

    Setting a minimum MSI in pipeline

    While developing a new feature, on a small scope

    Ok, I have the tools, now what?

    Tools are never the final answer!

    Release frequently

    Monitor

    Take educated risks!

    Learn from failures

    THANK YOU!

    Antonello D'Ippolito | twitter.com/antodippo | antodippo.com Try Infection and other testing tools here:
    https://github.com/antodippo/php-testing-playground

    Feedback!