Assertion failure during setup swallows test outcome

[Jean85]

Q	A
PHPUnit version	10.3.2
PHP version	8.2.8
Installation Method	Composer

Summary

Assertion failures during setUp lead to no test outcome, nor any indication of failure, only "No tests executed".

Current behavior

$ vendor/bin/phpunit
PHPUnit 10.3.2 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.2.8
Configuration: /var/www/pagamenti/phpunit.xml.dist

No tests executed!

How to reproduce

class FooTest extends TestCase
{
    public function setUp(): void
    {
        $this->assertTrue(false);
    }

    public function test(): void
    {
        $this->assertTrue(true);
    }
}

This is a partial redacted log:

# ...
Event Facade Sealed
Test Runner Started
Test Suite Sorted
Test Runner Execution Started (101 tests)
Test Suite Started (/var/www/***/phpunit.xml.dist, 101 tests)
Test Suite Started (integration, 101 tests)
Test Suite Started (Tests\FooTest, 1 test)
Test Preparation Started (Tests\FooTest::testBar)
Assertion Succeeded (Constraint: is not null, Value: Symfony\Component\HttpClient\TraceableHttpClient Object #3202 (
    # ...
))
Assertion Succeeded (Constraint: is not null, Value: Notifiche\Config Object #3179 (
    # ...
))
Assertion Failed (Constraint: is equal to 200, Value: 500)
Before Test Method Called (Tests\FooTest::setUp)
Before Test Method Finished:
- Tests\FooTest::setUp
Test Failed (Tests\FooTest::testBar)
Failed asserting that 500 matches expected 200.
After Test Method Called (Tests\FooTest::tearDown)
After Test Method Called (Tests\FooTest::tearDownProphecy)
After Test Method Finished:
- Tests\FooTest::tearDown
- Tests\FooTest::tearDownProphecy
Test Suite Finished (Tests\FooTest, 1 test)
# ...
Test Suite Finished (integration, 101 tests)
Test Suite Finished (/var/www/pagamenti/phpunit.xml.dist, 101 tests)
Test Runner Execution Finished
Test Runner Finished
PHPUnit Finished (Shell Exit Code: 1)

Expected behavior

Any indication on the failure.

[sebastianbergmann]

This breaks for me in a different way ;-)

Issue5493Test.php

<?php declare(strict_types=1);
namespace PHPUnit\TestFixture\Issue5493;

use PHPUnit\Framework\TestCase;

final class Issue5493Test extends TestCase
{
    protected function setUp(): void
    {
        $this->assertTrue(false);
    }

    public function testOne(): void
    {
        $this->assertTrue(true);
    }
}

$ phpunit --do-not-cache-result --no-configuration Issue5493Test.php
PHPUnit 10.3.4-10-g3bc96df835 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.2.10

There was 1 failure:

1) PHPUnit\TestFixture\Issue5493\Issue5493Test::testOne
Failed asserting that false is true.

/usr/local/src/phpunit/tests/end-to-end/regression/5493/Issue5493Test.php:11

No tests executed!

As you can see in the output above, I see the failure from the assertion in setUp(). But in the end the wrong No tests executed! statement is printed.

While this should be fixed, I would like to point out that assertions should not be used in a "before test method". These methods are only intended to be used to set up the test fixture. They should not be used to perform assertions.