Python multithreading for writing log

Question

I have an HTTP Server class, supporting multithreading, that while writing back the result to the client, it writes some logs (Logging module) as well. It slows down the request, and since the POST/GET methods returns only after the 'return' occures, I have to write the log before i send the result. I wish to do it by opening another thread that will receive the message, and write the log. I wish to have my Logging object as 'static', or a class member which is one and only, and is being shared to all threads that can acquire a lock on it and use it.

import logging

LOG_FILENAME = 'babyServer.log'
class Logger(object):
    logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s %(levelname)-8s %(message)s',
                    datefmt='%a, %d %b %Y %H:%M:%S',
                    filename='serverLog.log',
                    filemode='wa')


    @classmethod
    def Write(cls,level,log):
        level = level.lower()
        if level == 'error':
            logging.error(log)
        if level == 'info':
            logging.info(log)
        if level == 'exception':
            logging.exception(log)

So from somewhere outside of this file, I have currently a call such as Logger.Write('info', "some info") Instead, I wish that this call will actually open a new thread, that owns a reference to the single Logging I configured in the Logger class.

Roland Smith · Accepted Answer · 2013-08-20 18:31:11Z

3

Starting a thread also has overhead.

So I would suggest that you write a module that has three public functions;

start(filename) to start a logging thread
write(txt) to write to the log file.
stop() to stop the logging thread.

The write function places the text to log on the FIFO Queue and returns to the calling thread.

The logging thread reads items from the Queue and writes them to the logfile.

Say you have a file log.py like this;

"""Module to log to a file."""

import Queue
import threading

_name = None
_lt = None
_q = None
_stop = False


def _run():
    while not _stop:
        txt = _q.get()
        with open(_name, 'a+') as logfile:
            logfile.write(txt)


def start(name):
    global _lt, _q, _name
    if _name:
        raise RuntimeError('Logging already initialized')
    _name = name
    _q = Queue.Queue()
    _lt = threading.Thread(target=_run)
    _lt.start()


def stop():
    global _stop
    _stop = True
    _lt.join()


def write(txt):
    _q.put(txt, block=False)

Using the logging module instead of simply wrting to a file is left as an exercise for the reader.

edited Aug 20, 2013 at 18:31

answered Aug 20, 2013 at 17:48

Roland Smith

43.8k3 gold badges69 silver badges98 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

buddy123 Over a year ago

Can you please further explain (using my original code) How to maintain mutual variables from various threads with locking mechanism? Either by queue or by mutex. Im not sure how it goes with shared variables

buddy123 Over a year ago

What avoids _lt from being instantiated every time when a new write occures? It seems to me that its being recreated every time again and again

Roland Smith Over a year ago

The _lt thread is only created and started in the start() function, which can only be called once. If you try to call start() again, it will raise an exception. The only thing that write() does is put something in the queue, nothing more.

buddy123 · Accepted Answer · 2013-08-20 17:43:48Z

I found a very simple solution, using thread module.

import thread

Then, When I need to call my logging function, I simply use it with the following:

thread.start_new_thread( logging.error, (log,) )

where in this example, logging.error is a builtin function of logging, making an error log, and (log,) is a tuple (as it is required by the interface) of my the log message to write. It, as I expected, improve performance dramatically

    @classmethod
def Write(cls,level,log):
    level = level.lower()
    if level == 'error':
        thread.start_new_thread( logging.error, (log,) )
    if level == 'info':
        thread.start_new_thread( logging.info, (log,) )
    if level == 'exception':
        thread.start_new_thread( logging.exception, (log,) )

Collectives™ on Stack Overflow

Python multithreading for writing log

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related