Python logging overhead

Question

I'm writing my code using asyncio module with a lot of info/debug logging (logging.FileHandler). I'm concerned that extensive using of logging in asyncio will decrease the performance of my application, because logging is blocking operation.

What is the best solution for that? Didn't find any information on logging overhead.

Maybe using SocketHandler or MemoryHandler will help? For metrics I'm using statsd (which is also blocking operation, but must be very fast), but I'm more concerned with non-blocking logging.

Code example:

@asyncio.coroutine
def creator_worker(self):
    while not self.q.empty():
        with (yield from self.semaphore):
            sample = yield from self.q.get()
            logging.debug('Got new sample, processing')
            # start processing

I'm aware of

import os
os.environ['PYTHONASYNCIODEBUG'] = '1'
import asyncio

But that's not what I'm looking for.

Thank you

Leric · Accepted Answer · 2017-05-03 08:29:51Z

5

Logging document provided a solution, log through a QueueHandler, then use a QueueListener to handle log record in another thread. https://docs.python.org/3.6/howto/logging-cookbook.html#dealing-with-handlers-that-block

answered May 3, 2017 at 8:29

Leric

3133 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Andrew Svetlov · Accepted Answer · 2016-02-29 13:42:03Z

3

You may create a custom log handler. The handler will push logging record into a queue. Another thread may read data from the queue and perform actual logging.

As an option you may use SysLogHandler -- it's pretty fast. But, anyway, logging too much reduces performance, you can do nothing with it.

answered Feb 29, 2016 at 13:42

Andrew Svetlov

17.5k8 gold badges70 silver badges72 bronze badges

Collectives™ on Stack Overflow

Python logging overhead

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related