8

I would like to generate a new log file on each iteration of a loop in Python using the logging module. I am analysing data in a for loop, where each iteration of the loop contains information on a new object. I would like to generate a log file per object.

I looked at the docs for the logging module and there is capability to change log file on time intervals or when the log file fills up, but I cannot see how to iteratively generate a new log file with a new name. I know ahead of time how many objects are in the loop.

My imagined pseudo code would be:

import logging

for target in targets:
    logfile_name = f"{target}.log"
    logging.basicConfig(format='%(asctime)s - %(levelname)s : %(message)s',
                        datefmt='%Y-%m/%dT%H:%M:%S',
                        filename=logfile_name,
                        level=logging.DEBUG)


    # analyse target infomation
    logging.info('log target info...')

However, the logging information is always appended to the fist log file for target 1.

Is there a way to force a new log file at the beginning of each loop?

Improve this question

5 Answers 5

Reset to default
10

Rather than using logging directly, you need to use logger objects. Go thorough the docs here.

Create a new logger object as a first statement in the loop. The below is a working solution.

import logging
import sys


def my_custom_logger(logger_name, level=logging.DEBUG):
    """
    Method to return a custom logger with the given name and level
    """
    logger = logging.getLogger(logger_name)
    logger.setLevel(level)
    format_string = ("%(asctime)s — %(name)s — %(levelname)s — %(funcName)s:"
                    "%(lineno)d — %(message)s")
    log_format = logging.Formatter(format_string)
    # Creating and adding the console handler
    console_handler = logging.StreamHandler(sys.stdout)
    console_handler.setFormatter(log_format)
    logger.addHandler(console_handler)
    # Creating and adding the file handler
    file_handler = logging.FileHandler(logger_name, mode='a')
    file_handler.setFormatter(log_format)
    logger.addHandler(file_handler)
    return logger


if __name__ == "__main__":
    for item in range(10):
        logger = my_custom_logger(f"Logger{item}")
        logger.debug(item)

This writes to a different log file for each iteration.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

I've marked this as the answer as it gives the cleanest solution of the three presented at the time. Thanks very much for your help on this. It works perfectly.
Will a logger function defined this way be caught by logger = logging.getLogger() if I want to use it to log different modules? What if I want to create a file for each item?
@muammar, to reuse this, you need to pass this logger object to the desired class. Here, we are creating a logger for item .
@Mithilesh_Kunal I solved it as described below stackoverflow.com/a/56409646/1995261. Thanks for your reply!
4

This is not necessarily the best answer but worked for my case, and just wanted to put it here for future references. I created a function that looks as follows:

def logger(filename, level=None, format=None):
    """A wrapper to the logging python module

    This module is useful for cases where we need to log in a for loop
    different files. It also will allow more flexibility later on how the
    logging format could evolve.

    Parameters
    ----------
    filename : str
        Name of logfile. 
    level : str, optional
        Level of logging messages, by default 'info'. Supported are: 'info'
        and 'debug'.
    format : str, optional
        Format of logging messages, by default '%(message)s'.

    Returns
    -------
    logger
        A logger object.
    """

    levels = {"info": logging.INFO, "debug": logging.DEBUG}

    if level is None:
        level = levels["info"]
    else:
        level = levels[level.lower()]

    if format is None:
        format = "%(message)s"

    # https://stackoverflow.com/a/12158233/1995261
    for handler in logging.root.handlers[:]:
        logging.root.removeHandler(handler)

    logger = logging.basicConfig(filename=filename, level=level, format=format)

    return logger

As you can see (you might need to scroll down the code above to see the return logger line), I am using logging.basicConfig(). All modules I have in my package that log stuff, have the following at the beginning of the files:

import logging
import other stuff

logger = logging.getLogger()


class SomeClass(object):
    def some_method(self):
        logger.info("Whatever")
        .... stuff

When doing a loop, I have call things this way:

if __name__ == "__main__":
    for i in range(1, 11, 1):
        directory = "_{}".format(i)

        if not os.path.exists(directory):
            os.makedirs(directory)
        filename = directory + "/training.log"
        logger(filename=filename)

I hope this is helpful.

Improve this answer

Comments

3

This might not be the best solution, but it will create new log file for each iteration. What this is doing is, adding a new file handler in each iteration.

import logging
targets = ["a", "b", "c"]
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

for target in targets:
    log_file = "{}.log".format(target)
    log_format = "|%(levelname)s| : [%(filename)s]--[%(funcName)s] : %(message)s"
    formatter = logging.Formatter(log_format)

    # create file handler and set the formatter
    file_handler = logging.FileHandler(log_file)
    file_handler.setFormatter(formatter)

    # add handler to the logger
    logger.addHandler(file_handler)

    # sample message
    logger.info("Log file: {}".format(target))
Improve this answer

Comments

2

I'd like to slightly modify @0Nicholas's method. The direction is right, but the first FileHandler will continue log information into the first log file as long as the function is running. Therefore, we would want to pop the handler out of the logger's handlers list:

import logging
targets = ["a", "b", "c"]
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
log_format = "|%(levelname)s| : [%(filename)s]--[%(funcName)s] : %(message)s"
formatter = logging.Formatter(log_format)

for target in targets:
    log_file = f"{target}.log"
    
    # create file handler and set the formatter
    file_handler = logging.FileHandler(log_file)
    file_handler.setFormatter(formatter)

    # add handler to the logger
    logger.addHandler(file_handler)

    # sample message
    logger.info(f"Log file: {target}")
    # close the log file
    file_handler.close()
    # remove the handler from the logger. The default behavior is to pop out      
    # the last added one, which is the file_handler we just added in the 
    # beginning of this iteration.
    logger.handlers.pop()
Improve this answer

Comments

0

Here is a working version for this problem. I was only able to get it to work if the targets already have .log before going into the loop so you may want to add one more for before going into targets and override all targets with .log extension

import logging

targets = ["a.log","b.log","c.log"]
for target in targets:

    log = logging.getLogger(target)
    formatter = logging.Formatter('%(asctime)s - %(levelname)s : %(message)s', datefmt='%Y-%m/%dT%H:%M:%S')
    fileHandler = logging.FileHandler(target, mode='a')
    fileHandler.setFormatter(formatter)
    streamHandler = logging.StreamHandler()
    streamHandler.setFormatter(formatter)
    log.addHandler(fileHandler)
    log.addHandler(streamHandler)

    log.info('log target info...')
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.