Dynamic filepath & filename for FileHandler in logger config file in python

Question

I have a python log config file with a filehandler of the below form.

[handler_filelog]
class: FileHandler
args = ('/var/tmp/log/client.log','a')

Instead, I need it in the below form (dynamically generated path).

[handler_filelog]
class: FileHandler
args = ('/var/tmp/log_<unique_string>/client.log','a')

Multiple instances of the program may be running and hence non-conflicting log paths and files are to be used. The logger once setup need not change till end of program execution.

Is there a way to handle this using the config file approach? I am not keen on resorting to creating the loggers/handlers/formatters by myself since my log config file has many of these and config file based approach is much nicer.

(Update: I am using python 2.4)

Yes. As commented below it works if I instantiate the handler myself. To do the same via config file, I haven't figured out how to make the definition of my class visible to logging module. — Gns
– Gns, Commented Feb 25, 2012 at 0:41
Were you able to test the updated code posted in the answer? It worked for me with the old config format. So you don't need a dictionary based config file to get it to run. — Phani
– Phani, Commented Mar 1, 2012 at 21:28

Phani · Accepted Answer · 2012-02-25 02:36:33Z

14

This does what you need. You should first extend the FileHandler class. Place this in a file, say myHandler.py in your config file's directory:

import logging
import random
import os
class myFileHandler(logging.FileHandler):
    def __init__(self,path,fileName,mode):
        r = random.randint(1,100000)
        path = path+"/log_"+str(r)
        os.mkdir(path)
        super(myFileHandler,self).__init__(path+"/"+fileName,mode)

And then in the config file, you can use this custom FileHandler like this

class: myHandler.myFileHandler
args = ('/var/tmp','client.log','a')

I tested this one on my machine

edited Feb 25, 2012 at 2:36

answered Feb 23, 2012 at 6:59

Phani

3,3354 gold badges29 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Gns Over a year ago

With extending of FileHandler, I was able to get the behavior by creating my own handler and adding to the logger. But could not get this to work via the config file. The "randomFileHandler" is not defined at the point where logging module is using config ini. Am checking if there is a easy way to make the class definition visible or if I end up going the way of own handlers.

Gns Over a year ago

I attempted to make the definition visible by defining logging.<myhandler> = <myhandler>, but I run into the problem of AttributeError: MyFileHandler instance has no attribute 'level' -> possibly bugs.python.org/issue8165. I am still checking if it is my incorrect usage or a known bug of logging module.

Phani Over a year ago

I am new to python inheritance. So, I was doing something terrible with the subclass initially. Looks like it works now though...

user59634 Over a year ago

Thanks for your answer. I had a different requirement -- to get the name of the file to which the logger was logging to. Just added a method to the class as suggested by you and it works great. Thanks a lot!

Matan Tubul Over a year ago

how can i use in your answer os.environ passed to the args attribute?

Vinay Sajip · Accepted Answer · 2012-02-25 18:30:32Z

6

If you're using Python 2.7 or 3.2, you can use dictionary-based configuration, which allows you to specify arbitrary callables to return handlers - you could e.g. use the process PID to construct the file name.

Update: If you are using 2.4, you can use the logutils package, which should work with Python 2.4 (apart from the LoggerAdapter class, which requires 2.5 or later). The logutils package contains the dictionary-based configuration functionality.

edited Feb 25, 2012 at 18:30

answered Feb 23, 2012 at 22:27

Vinay Sajip

100k15 gold badges184 silver badges196 bronze badges

3 Comments

Gns Over a year ago

I am using python 2.4. Thanks for the hint though, if the alternatives are not simple enough, I will probably remove config file and implement a dictionary based configuration myself. As of now this path is not yet desirable.

Gns Over a year ago

Thanks for the update pointing to logutils. This is a useful hint. This does address the question but I picked the other answer since it is more suitable for my needs (does not introduce new library dependency).

Kenji Noguchi Over a year ago

This answer should get more up-vote. Sorry for the op that s/he has to use Python2.4 but logging.config.dictConfig is the current standard where you can easily manipulate the config.

王国梁 · Accepted Answer · 2018-01-18 05:38:45Z

"CallbackFilter" can be used to implement Dynamic filepath & filename for in logger config file in python. You can define write_dynamic_log as below:

def write_dynamic_log(record):
    now = datetime.datetime.now().strftime("%Y%m%d%H%M%S")
    dynamic_log_name = '/var/log/test_%s.log' %now
    log_file = open(dynamic_log_name, "w")
    log_file.write(record.msg)
    log_file.close();
    return True

Then in the config file, you can use this filter like this:

[handler_filelog]
class: FileHandler
formatter: brief
level   : INFO
filters: [write_dynamic_log]
filename: static.log

The INFO or above log will be output to static.log and also to dynamic_log.

I tested it in my django project, in which I wrote config in my settings.py. It works fine. LOGGING will is like:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '[%(levelname)s] %(asctime)s - %(pathname)s:%(lineno)d\n%(message)s'
        },
        'debug': {
            'format': '[%(levelname)s] %(asctime)s - %(pathname)s:%(lineno)d\n\033[34m%(message)s\033[0m'
        },
        'error': {
            'format': 'Component: %(module)s\nErrorCode: %(funcName)s\nReason: %(funcName)s\nDetail: [%(message)s]'
        },
    },
    'filters': {
        'write_error_logs': {
            '()': 'django.utils.log.CallbackFilter',
            'callback': write_error_log,
        }
    },
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'formatter': 'verbose',
        },
        'debug': {
            'class': 'logging.StreamHandler',
            'formatter': 'debug',
        },
        'error': {
            'level': 'ERROR',
            'class': 'logging.FileHandler',
            'filename': '/var/log/SmartStorageApp.err',
            'formatter': 'error',
            'filters': ['write_error_logs'],
        },
    },
    'loggers': {
        'django': {
            'handlers': ['debug' if DEBUG else 'console', 'error'],
            'level': 'INFO',
            'propagate': True,
        },
    }
}

Collectives™ on Stack Overflow

Dynamic filepath & filename for FileHandler in logger config file in python

3 Answers 3

5 Comments

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

5 Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related