Python logging - How to add a new log file for each iteration in a loop?

Question

I am trying to generate new log file for each iteration in a loop using:

def custlogger(Filename,loglevel=logging.DEBUG):
    # set method name from where its called
    logger_name = inspect.stack()[1][3]
    # create logger
    logger = logging.getLogger(logger_name)
    logger.setLevel(loglevel)
    # create console handler of file handler and set log level
    file_handler = logging.FileHandler(filename=Filename)
    # create formatter
    formatter = logging.Formatter('%(asctime)s,%(msecs)d %(name)s %(levelname)s %(message)s')
    # add formatter to console or file handler    
    file_handler.setFormatter(formatter)
    # add console handler to loogger
    logger.addHandler(file_handler)

and I call the newlogger = custlogger(newfilename) and shut down logging at the end of each iteration logging.shutdown() inside the loop. Is this the right way to do it?

Here is an example how I am using it:

def analyze_scds_files(scds_directory, sample_name):
    logfile = scds_directory+'/log_'+sample_name+'.txt'
    newlogger = custlogger(logfile,logging.ERROR)                                                        
    try:
        dosomething()
    except Exception as e:
        newlogger.logger.exception('error while trying to ...: ')
        sys.exit()
    .
    .
    .
    # Shut down the logger
    logging.shutdown()

    return status

Would you agree to show us a basic version of the using code with the for loop and so on? — Floh
– Floh, Commented Apr 25, 2022 at 15:39
You probably want to use logging.handlers.RotatingFileHandler (or a custom subclass thereof). Rather than using logging.shutdown (which almost never needs to be called explicitly; it's meant to be called before a script exits, and the module configures itself to do so), you just call the logger's doRollover method to switch to the next file. — chepner
– chepner, Commented Apr 25, 2022 at 15:46
Move the example to the question itself. It can be in a code block and will be more readable there. — tdelaney
– tdelaney, Commented Apr 25, 2022 at 15:49
@chepner, do I need to call it at the end of each loop that I am using the handler? — MArablu
– MArablu, Commented Apr 25, 2022 at 15:50
Yes; rotating file handlers are commonly used to automatically switch log files after a certain period of time has passed or after the file reaches a certain size, but you can manually switch to the next file any time you like. — chepner
– chepner, Commented Apr 25, 2022 at 15:52

Marc · Accepted Answer · 2022-04-25 15:43:05Z

2

Suggestions, a little under documented question, Move this out of func and call in header (1x, you are redeclaring)

logger = logging.getLogger(logger_name)
logger.setLevel(loglevel)

The minimal pattern is:

import logging
logger = logging.getLogger(__name__)
# in header ^^ in loop ---v
logger('xx') # call this in loop

Get it to print in terminal first, then add formatting and other features. KISS v1, then complify.

answered Apr 25, 2022 at 15:43

Marc

1,99719 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python logging - How to add a new log file for each iteration in a loop?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related