Python proper logging with filehandlers

Question

I am using python logging in my django application. A class that connects to a backend api initialises this logger with a filehandler if needed. The class gets instantiated everytime an api call is made. I have tried making sure additional handlers are not added every time, but

lsof | grep my.log

shows an increasing amount of handlers on my log file and after a while my server fails due to this open file limit.

 self.logger = logging.getLogger("FPA")

        try:
            if self.logger.handlers[0].__class__.__name__=="FileHandler":
                pass
        except Exception, e:
            print 'new filehandler added'+str(e)
            ch = logging.FileHandler(FPA_LOG_TARGET)
            formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(message)s - %(pathname)s @ line %(lineno)d")
            ch.setFormatter(formatter)
            self.logger.setLevel(logging.DEBUG)
            self.logger.addHandler(ch)

I realise this may not be the best way to do this, but I have not found the error in my implementation so far.

what does print 'new filehandler added'+str(e) print out in your log statement? — Mahmoud Abdelkader
– Mahmoud Abdelkader, Commented Mar 16, 2011 at 10:18
print 'new filehandler added'+str(e) says index out of range, so even if i already initialised the logger, a next instance of my class will not see any handlers. — endre
– endre, Commented Mar 16, 2011 at 10:58
You need to provide more context for this. It looks totally like the wrong approach, but we can't tell where in your code this occurs? A view function? settings.py? urls.py? Further, why do you keep adding new file handlers? Why isn't this declared once in a logging.ini file? — S.Lott
– S.Lott, Commented Mar 16, 2011 at 12:32

Jerzyk · Accepted Answer · 2011-03-21 13:46:10Z

I did not analysed if for a long time, but it looks like a concurrency problem.

Each process/thread is keeping it's own list of the file handles to the opened log files.

How to fix it? For the multithreaded code, make sure that there is a global dictionary where all handles are kept. For the multiprocess - I'm afraid I do not have an answer... each process is keeping it's own file-handles, maybe mapping it to the memory (memory mapped files could be an option), but I'm not sure that this is good solution - see this remark.

But the main question is why do you need to do such a thing.

First of all, you can use logging.conf file to initialize all your loggers/handlers/formatters and when needed (e.g. specific loger is extensive and you want to log it to the separate file) add another logger with different filename. Which is quite sensible if you will add one logger per django app, by adding in the main __init__.py of the app:

import logging
log = logging.getLogger(__name__)

and then import log in the rest of the app code (views, models, etc.)

To use logging.conf add following lines to your settings.py:

import os
import logging
DIRNAME = os.path.abspath(os.path.dirname(__file__))
logging.config.fileConfig(os.path.join(DIRNAME, 'logging.conf'))

Yes, it is manual, but you do not need to change code, but simply a config file.

Another approach (if you really want to have one file per logger type) is to have a separate process which will keep files open, accept connections from the application. Logging module documentation has a nice example of this method.

Last, but not least there are already some nice solutions which may be helpful. One, quite good, is to use django-sentry. This module can log all your exceptions, 404 (with extra middleware - included) and capture all logging (via included logging handler).

Provided UI will gave you a possibility to search all the logged messages, filter them by the severity and logging source. But this is not limited to those - you can simply add your own modules.

Paul Painter · Accepted Answer · 2025-04-17 14:49:14Z

0

For the sake of later searchers, like me, although it is 11 years later, Python now has the logging.handlers.QueueHandler class which supports background threading for log message processing. It is thread-safe even when used by the applications own threads. I found it very easy to learn and use.

Currently, using background QueueHandlers is better practice, particularly for apps that do heavy logging with JSON log formatting. All processing of log formatters is done in a thread so as to not suppress the calling app thread. Good luck.

answered Apr 17 at 14:49

Paul Painter

11 silver badge1 bronze badge

Collectives™ on Stack Overflow

Python proper logging with filehandlers

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related