Is there a simple way to invert a test or test group?

Question

Assume you have written a failing test to exploit a bug. You want to push this to upstream and then create a story that is about fixing the code so that the test passes. However, this is prevented by the pipeline that runs all the test.

I came up with a solution to this: Add the test to a group, and then edit phpunit.xml to exclude that group, making the pipeline skip that test. After the bug is solved, you simply remove the test from the excluded group.

A problem here is that the last step can be forgotten. The pipeline will pass regardless if the test passes or not.

I figured that a solution to this would be to instead of excluding a test, we expect it to fail. Is there a way to configure phpunit.xml in such a way that all tests that belongs to a specific group are expected to fail at least one assertion? I have looked in the documentation, but I have not found anything. Same thing with annotations. A solution would be if there was an annotation like @expectToFail but I didn't find anything like that either.

Maybe inverting tests is not the correct approach. But to write a short summary of what I want to do without assuming anything about the best way to do it.

1. I should be able to write a failing test that MUST NOT make the pipeline fail.
2. If the code that is tested by the test gets modified so that the test in (1) passes without any other changes to the project, the pipeline MUST fail.
3. When (2) has happened, there MUST be a very simple way to invert this, so that the pipeline MUST fail when the test fails, and MUST NOT fail when the test passes.

If possible, I want to do this without altering the test code, for instance by manually changing assertFalse to assertTrue.

I might add that it is bitbucket pipelines

To clarify, here is basically what I want. Let's say we have this code:

class Foo {
    // Function with obvious bug
    public function sum(a, b) {
        return a-b;
    }
}

Obviously, the function sum has a bug. So we write a unit test:

class FooTest {

    /**
     *  @test
     */
    public function bugExploit() {
        $foo = new Foo();
        static::assertEqual(8, $foo->sum(3,5));
    }
}

Now we commit this test and push it. The pipeline should pass after this push, provided that it passed before. Then we write a story about solving this bug.

Time for next developer. They pick up the story that says "Modify Foo:sum so that the test FooTest::bugExploit passes". After having read this, the developer changes return a-b to return a+b, and changes nothing else. They push the code. Now I expect the pipeline to fail.

A viable solution to this would be if there existed an annotation called @fail that the first developer adds to the doc comment in order to expect the test to fail, and then the second developer removes that line.

Answer 1

UPDATE

Not sure what you're tryin to do would be consider a good practice, by the way i'll try to answer you point by point:

1. This behavior could be obtained in many ways: using markTestIncomplete(), using (and excluding) groups/levels/paths etc. or using a try/catch statement. Note that, if you use a single try/catch for multiple assertions, the first fail will block the whole test. To be sure to test all the assertions, you need to make multiple try/catch statements:

function testPassWithFailures() {
    $failures = [];
    try {
        // First assertion
        $this->assertTrue(false);
    } catch(PHPUnit_Framework_ExpectationFailedException $e) {
        $failures[] = $e->getMessage();
    }
    try {
        // Second assertion
        $this->assertTrue(false);
    } catch(\PHPUnit\Framework\ExpectationFailedException $e) {
        $failures[] = $e->getMessage();
    }

    // ...

    if(!empty($failures))
    {
        // Here you can put your logic and decide if failures should make the test fail
        // or pass

        // Throw a new exception to make it fail
        throw new \PHPUnit\Framework\ExpectationFailedException (...);
    }
}

As you can see, it's a not-so-good approach, but maybe it could help you to think to different solutions.

2/3. This is the hard part. What do you mean by "without any other changes to the project"? You just want to consider edits to the current Test file? What happens when you perform a rebase or a merge? Assuming that you already know issues of this approach, you can try this particular approach:

First, create a testsuite based on file suffix (ex. TestDemoIncomplete.php)

<testsuites>
    <testsuite name="incomplete-tests">
        <directory suffix="Incomplete.php">test</directory>
    </testsuite>
</testsuites>

Then, you could create a script that runs your test and perform a git diff, to understand which files were modified, and then runs phpunit on incomplete-tests. Then, you have to put some logic to your script to remove the "Incomplete" suffix when tests pass. This way, when your test pass for the first time, its "Incomplete" mark would be removed and this will be considered as a standard test.

You can build this script to run on a Git Hooks or directly inside your pipeline (but, as you will change the file name, a new commit would probably be required).

I'm not suggesting you an out-of-the-box solution, i'm trying to give you hints on what you want to achieve. In my opinion, this is a bad approach and i would try to find a solution using markTestIncomplete as it exists for this kind of cases (also, you can parse the output of the script to get warnings and then use this information to block your pipelines or to simply warn developers).

OLD ANSWER

You can mark your tests as incomplete (https://docs.phpunit.de/en/10.3/writing-tests-for-phpunit.html#incomplete-tests):

$this->markTestIncomplete(
    'This test has not been implemented yet.',
);

Tests will pass, but a warning will be generated in the output:

...

There was 1 incomplete test:

1) WorkInProgressTest::testSomething
This test has not been implemented yet.

/path/to/tests/WorkInProgressTest.php:12

OK, but there were issues!
Tests: 1, Assertions: 1, Incomplete: 1.

Answer 2

Update

This answer does not work. I'm working on a solution. The problem is that the pipeline will pass as long as at least one of the tests tagged @failing fails. I want it to require that ALL of them fails in order to pass.

Old

I managed to solve it.

First, I configured phpunit.xml to exclude tests with the annotation @group failing, as I already mentioned in the question. I did this with:

<groups>
    <exclude>
        <group>failing</group>
    </exclude>
</groups>

Then I copied the test command in the pipeline. It's on the form docker run testuser ./vendor/bin/phpunit. My idea here was to simply negate the return code. I tried several methods that worked in a terminal, but I never got the bitbucket pipeline to understand the negation.

So instead I edited the Makefile of the project, and added this:

failing-tests:
    ! docker run testuser ./vendor/bin/phpunit --group failing

Note the ! in front of the command. That's the magic.

And after that I simply added make failing-tests in bitbucket-pipelines.yml

Tried this with a simple test on the form

class MyTest extends TestCase{
    /**
     * @test
     * @failing
     */
     public function aFailingTest(): void {
         static::assertTrue(false);
     }
}

The main test suite ignored this test thanks to the annotation, and the pipeline passed. When I changed to assertTrue(true) the pipeline failed. Exactly what I wanted.