Why JUnit tests using ByteArrayOutputStream fail when running multiple tests but pass when run individually?

Question

I'm currently working with JUnit tests that use ByteArrayInput stream to give input to my class and BytearrayOutputStream to grab the output and compare it whit and expected output. When tests are run individually, they pass just fine, but when testing the whole test class, only the first one passes and the subsequent ones fail because there is nothing in the OutputStream.

Here is the class under test:

package org.oscargs;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

import java.io.InputStream;
import java.io.PrintStream;
import java.nio.charset.StandardCharsets;
import java.util.Scanner;


@FunctionalInterface
interface Function {
    void apply();
}
public class Menu {
    private Function[] players;
    private Logger logger;
    private Scanner scan;


//    private final PrintStream printStream;

    Menu() {
        logger = LogManager.getLogger();
        players = new Function[2];
    }

    public void selectInputStream(final InputStream inputStream) {
        this.scan = new Scanner(inputStream, StandardCharsets.UTF_8);
    }

    public void teardown() {
        this.scan = null;
        this.logger = null;
        this.players = null;
    }

    public void showMenu() {
        logger.info("Input command: ");
        String input = scan.nextLine();
        String[] parameters = input.split(" ");

        if (parameters.length == 1) {
            if (!parameters[0].equals("exit")){
                logger.info("Bad parameters! Only available parameters are 'start <player1> <player2>' and 'exit'\n");
                return;
            }

            logger.info("Bye\n");
//            Main.endGame();
//            System.exit(0);

        } else if (parameters.length == 3) {
            if (! parameters[0].equals("start")){
                logger.info("Bad parameters! Only available parameters are 'start <player1> <player2>' and 'exit'\n");
                return;
            }

            for (int i = 0; i < 2; i++) {
                char p = Constants.getPlayerChar(i);
                switch (parameters[i+1]) {
                    case "easy":
                        players[i] = () -> ComputerPlayer.moveEasy(p, Constants.EASY);
                        break;
                    case "medium":
                        players[i] = () -> ComputerPlayer.moveMedium(p);
                        break;
                    case "hard":
                        players[i] = () -> ComputerPlayer.moveHard(p);
                        break;
                    case "user":
                        players[i] = () -> Player.move(p);
                        break;
                    default:
                        logger.info("Bad parameters!");
                        return;
                }
            }

            Game.init();

            while(Game.continueGame) {
                Game.game(players[0], players[1]);
            }

        } else {
            logger.info("Bad parameters!");
        }
    }
}

Here is my test class

package org.oscargs;
import org.junit.jupiter.api.*;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.junit.jupiter.MockitoExtension;

import java.io.*;

import static org.junit.jupiter.api.Assertions.*;

@ExtendWith(MockitoExtension.class)
class MenuTest {
    private Menu menu;
    private OutputStream outputStream;



    @BeforeEach
    void setUp() {
        try {
            this.outputStream = new ByteArrayOutputStream();
            this.outputStream.flush();
            System.setOut(new PrintStream(outputStream));
////        this.inputStream = new ByteArrayInputStream();
            this.menu = new Menu();
        } catch (IOException e) {
            throw new RuntimeException(e);
        }

    }

    @AfterEach
    void tearDown() {
        try {
            this.outputStream.flush();
            this.outputStream.close();
            this.outputStream = null;
            System.setOut(null);

//            this.inputStream.close();

            this.menu.teardown();
            this.menu = null;

        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

    @Test
    void badCommandLenghtOneTest() {
        // Given
        String expectedOutput = "";
        String input = "badcommand"+ System.lineSeparator();
        ByteArrayInputStream in = new ByteArrayInputStream(input.getBytes());

        menu.selectInputStream(in);
        menu.showMenu();

        String givenOutput = outputStream.toString();

        expectedOutput += "Input command: ";
        expectedOutput += "Bad parameters! Only available parameters are 'start <player1> <player2>' and 'exit'\n";

        assertEquals(expectedOutput, givenOutput);
    }

    @Test
    void exitCommandTest() {
        // Given
        String expectedOutput = "";
        String input = "exit"+ System.lineSeparator();
        ByteArrayInputStream in = new ByteArrayInputStream(input.getBytes());

        menu.selectInputStream(in);
        menu.showMenu();

        String givenOutput = outputStream.toString();

        expectedOutput += "Input command: ";
        expectedOutput += "Bye\n";

        assertEquals(expectedOutput, givenOutput);
    }
}

And here is the error log

-------------------------------------------------------------------------------
Test set: org.oscargs.MenuTest
-------------------------------------------------------------------------------
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.452 s <<< FAILURE! -- in org.oscargs.MenuTest
org.oscargs.MenuTest.badCommandLenghtOneTest -- Time elapsed: 0.008 s <<< FAILURE!
org.opentest4j.AssertionFailedError: 
expected: <Input command: Bad parameters! Only available parameters are 'start <player1> <player2>' and 'exit'
> but was: <>
    at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
    at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
    at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
    at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
    at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
    at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141)
    at org.oscargs.MenuTest.badCommandLenghtOneTest(MenuTest.java:64)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

I'm working with Java 17 and JUnit 5.10.0

<dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>5.10.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>5.10.0</version>
            <scope>test</scope>
        </dependency>

I've already tried closing the streams and setting them to null after the execution of each test to prevent any leakage and open stream that may interfere, and setting my scan and logger fields in the class under test to null, but it seems to have no effect.

Thanks in advance

Answer 1

It seems that all of the methods on your Game class are static so I'm sure you have some mutable state on the Game class too. Statics are like globals and they can "bleed" from one test to another. For example TestA might mutate the Game globals and then TestB reads the "dirty" game state.

I suggest you refactor the Game class so that there are no static methods or variables. Instead you should instantiate a new Game(player1, player2) and invoke methods on that instance. Each test could then have it's own instance of Game which would guarantee that state can't "bleed" from one test to another.

Answer 2

You are assuming that Log4j Logger writes directly on the System.out, which I don't think is the case.

In fact, I think that once the ConsoleLogger is created, changing the System.out has no effect.