How to output and log messages at the same time in python

Question

I want something like what the tee command does in Linux based systems, but in pure python code.

I have tried using this snippet:

import sys

# Define the file name for logging
log_file_name = 'output.log'

# Open the file in append mode
log_file = open(log_file_name, 'a')

# Function to log output to both console and file
def log(message):
    # Write to console
    print(message)
    # Write to file
    log_file.write(message + '\n')
    # Flush the buffer to ensure the message is written immediately
    log_file.flush()

# Redirect stdout and stderr to the log function
sys.stdout.write = log
sys.stderr.write = log


print("Hello world!")

However I get this error:

object address  : 0x7f88e6b1c1c0
object refcount : 2
object type     : 0x9e14c0
object type name: RecursionError
object repr     : RecursionError('maximum recursion depth exceeded')
lost sys.stderr

and whenever I comment out the print statement in the log function it logs the message into the log file but does not output the message into the console. Why is this happening and how can I fix it?

Answer 1

What goes wrong

print in python writes the given message to stdout. The reason you're getting this error, is that you've rebound sys.stdout.write to your log function, but inside this log function, there is also a call to print. The line print(message) now again calls stdout.write, which is the log function. This happens recursively until you hit the max recursion depth, which is 1000 by default in python.

Solution in python

I don't think you should want to do what you're currently doing. By overwriting sys.stdout.write, you're hacking your own logging into python, which may well break any packages you're using, or at the very least it will greatly confuse anyone you're working with.

It would probably be simplest to just write something like the following, and replace print statements with output statements in your code.

def output(message):
    # To console
    print(message)

    # To file
    with open(log_file_name, 'a') as file:
        file.write(message + '\n')

Note that you were also not closing the file after writing to it, potentially risking data corruption. See for instance this explanation for why with statements are your greatest friend!

But if possible, using tee probably is the simplest solution.

EDIT:

Note on the logging module

Actually, such functionality is also possible using the logging module from the standard library, for a more production-ready solution. The following configuration writes to both the console and a file every time the test logger gets used:

LOGGING = {
    "version": 1,
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
        },
        "file": {
            "class": "logging.FileHandler",
            "filename": "debug.log",
        },
    },
    "loggers": {
        "test": {
            "handlers": ["console", "file"],
        },
    },
}

Answer 2

Here is one way:

import sys

_print = print

def print(*args, **kwargs):
    _print(*args, **kwargs, file=open('output.log','w'))
    _print(*args, **kwargs)

print("Hello world!")

Explanation

print() function already has an argument for file destination. https://docs.python.org/3/library/functions.html#print

We preserved the original print function within _print() and call it from our print function to serve our both needs. This way, the calling code doesn't need to change.

Answer 3

The Problem:

The issue arises because the print statement also outputs to sys.stdout, causing the function to recursively call itself as it detects an output in the stdout stream.

Solution:

To resolve this, a LoggingWrapper class is created to retain the original stdout and stderr streams. This class modifies stdout and stderr to use the wrapper, ensuring that data printed to these streams is directed to the original streams and also logged into the log file.

import sys

class LoggingWrapper:
    def __init__(self, original_stream):
        self.original_stream = original_stream

    def write(self, message):
        # Write to the original stream
        self.original_stream.write(message)
        # Write to the log file
        log_file.write(message)
        # Flush the buffer to ensure the message is written immediately
        log_file.flush()

    def flush(self):
        # Ensure the original stream is flushed
        self.original_stream.flush()

# Define the file name for logging
log_file_name = 'output.log'

# Open the file in append mode
log_file = open(log_file_name, 'a')

# Create wrappers for stdout and stderr
stdout_wrapper = LoggingWrapper(sys.stdout)
stderr_wrapper = LoggingWrapper(sys.stderr)

# Redirect stdout and stderr to the wrappers
sys.stdout = stdout_wrapper
sys.stderr = stderr_wrapper

This solution ensures that messages are logged into the specified log file while maintaining the original behavior of stdout and stderr. It prevents recursion errors by separating the logging process from the standard output stream.