1

I'm definitely missing something when it comes to logging using the logging module. I can get it to work just fine if I have just 1 file with everything in it, but when I try to split things up into separate files or packages I'm not sure what to do.

Suppose I have a package with the following directory tree:

mypackage/
├─ __init__.py
├─ utils.py
├─ my_logger.py
main.py

In my package I have a logger (in my_logger.py) that is configured nearly to my liking. In my main.py file, I initialize that logger and add a FileHandler to start writing to a file.

However, I'd also like to log to this file from other files in my package. Say, for example, the file utils.py. How do I make the instance of logger in utils.py know about the FileHandler?

Suppose I have a series of functions in utils.py, say, myfunc1() and myfunc2(), and I'd like each to also log to the same file. I don't want to have to pass my instance of logger to each individual function.

There feels like an obvious setting I am missing. What should I have in which file in order to make sure they all know about the same logger instance everywhere in the package?

(A detail that may or may not be relevant — I've actually subclassed Logger to give it some custom behaviour. But this shouldn't affect the answer to this question, I don't think.)

Improve this question
9
  • The general idea for loggers is that the logger for individual modules should simply be set up using getLogger(__name__) and if you want specific functions for logging when executing your main program, that's the only location where the custom configurations be used (e.g. attaching a handler to log to specific file(s)). Another thread describing another similar setup. Commented Jan 6, 2023 at 1:17
  • So each file (module) in the file should make their own "getLogger" call to log to? What if I want to consolidate all the results from all the different files into a single log file? Commented Jan 6, 2023 at 1:19
  • They are not logged to different files - modules shouldn't do anything to the logger instance they created using getLogger(__name__) (note that's the only setup command, so they won't have handlers set up to write to their own files), they should leave it to the main program to set it up. It would be easier for you to actually try it out and see the results yourself. Commented Jan 6, 2023 at 1:20
  • 1
    They are not logged to a void, they are logged to the named logger as per the __name__ of the module, e.g. calling getLogger(__name__) inside mypackage/utils.py should return a logger named 'mypackage.utils', and logs will be written to it. In your main application you should get the root logger just by getLogger() and attach handlers to it, which will capture all the subloggers. If you want to only deal with messages in your package, simply getLogger('mypackage') which will capture all loggers underneath 'mypackage.*', which includes mypackage.utils. Commented Jan 6, 2023 at 1:23
  • 1
    Yes. getLogger('a') == getLogger('a') is True. Logging to a logger returned by getLogger('a.b') will show up for handlers attached to getLogger('a'). Commented Jan 6, 2023 at 1:35

1 Answer 1

Reset to default
2

Thanks to @metatoaster I have the solution I was looking for.

I have the files organized in the tree above.

In mypackage.my_logger.py:

import logging
import sys

logger = logging.getLogger('mypackage')
ch = logging.StreamHandler(sys.stdout)
logger.addHandler(ch)
logger.setLevel(logging.DEBUG)

In mypackage.utils.py:

import logging

logger = logging.getLogger('mypackage')

logger.info('Running in the body of mypackage.utils.py')


def myfunc():
    logger.info("Running in myfunc() in mypackage.utils.py")

In my main.py file:

from mypackage.my_logger import logging

from mypackage.utils import myfunc


logger = logging.getLogger('mypackage')

logger.info('Running in main.py')

myfunc()

The output is:

Running in the body of mypackage.utils.py
Running in main.py
Running in func() in mypackage.utils.py

Note how, by importing logging from mypackage.my_logger, I'm running the setup for the StreamHandler in the body of that module.

I understand now why the convention is to always get a logger that calls for __name__, as in getLogger(__name__). Explicitly naming the package everywhere works for my current usecase. Thanks again for the help!

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Yeah that's certainly an approach if you want to isolate your logging setup to a module that gets automatically set up if you import it, though my recommendation is to wrap that in a function to avoid side-effects - this prevents people who don't want random log files being generated just simply by importing your package being inconvenienced by what they deem as unwanted output. Your main program will import the function (instead of logging) and call it to set up the logger only upon running. Also reason for using the __name__ helps identify the module that logged the message.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.