Yet Another Python Logging Setup

Question

I've been reading on logging module in python and several blogs on how to set it up, however, no blog gave a scheme for more complex setup.

I would like:

Use external config file (see below).
Have external module to handle log settings (create log file w/mylogfile+datetime (see below).
Finally, instantiate a logger in a class to log from multiple methods to the output file.

Right now, admittedly, I've a mishmash of settings, and would like to have some pointers to clean this mess :-).

I believe the external config file loads up fine, but there is no log file created.

Example main:

#!/bin/env python

import os
import sys
import logging.config
from datetime import datetime

datetime.now().strftime('mylogfile_%H%M%d%m%Y.log')

LOG_CONFIG = '../config/logging.conf'
#logging.config.fileConfig(LOG_CONFIG)
#logger = logging.getLogger(datetime.now().strftime('mylogfile_%H%M%d%m%Y.log'))

   def setup_logger():
       logging.config.fileConfig(LOG_CONFIG)
       datetime.now().strftime('mylogfile_%H%M%d%m%Y.log')
       logger = logging.getLogger(datetime.now().strftime('mylogfile_%H%M%d%m%Y.log'))

class TestLog(object):
def __init__(self):
    self.logger = logging.getLogger(__name__)
    self.__sub_test = 0

def add_test(self):
    self.logger.debug('addition')
    a = 1 + 1
    self.logger.debug('result {}'.format(a, 1))

def sub_test(self):
    self.logger.debug('subtraction')
    b = 5 -2
    self.logger.debug('result {}'.format(b, 1))


def main():
    # 'application' code
   logger.debug('debug message')
   logger.info('info message')
   logger.warn('warn message')
   logger.error('error message')
   logger.critical('critical message')
   #setup_logger()
   test1 = TestLog()
   print test1.add_test()
   print test1.sub_test()

if __name__ == "__main__":
    sys.exit(main())

Conf file:

[loggers]
keys=root

[handlers]
keys=consoleHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=DEBUG
handlers=consoleHandler

[logger_sLogger]
level=DEBUG
handlers=consoleHandler
qualname=sLogger
propagate=0

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)

[handler_fileHandler]
class=FileHandler
level=DEBUG
formatter=fileFormatter
args=('%(logfilename)s',)

[formatter_fileFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
datefmt=

[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
datefmt=

Update based on user5359531 reply, I've modified the scripts to the ones below, but there is an issue with the file handler where a file isn't created and messages are not appended to the file.

utilityLogger:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

'''
My app
'''
# ~~~~~ LOGGING SETUP ~~~~~ #
# set up the first logger for the app
import os
import testLogging as vlog
# path to the current script's dir
scriptdir = os.path.dirname(os.path.realpath(__file__))

LOG_CONFIG = '../config/logging.conf'
print scriptdir

def logpath():
     '''
    Return the path to the main log file; needed by the logging.yml
    use this for dynamic output log file paths & names
    '''
    global scriptdir
    return (vlog.logpath(scriptdir = scriptdir, logfile = 'log.txt'))

logger = vlog.log_setup(config_file=LOG_CONFIG, logger_name="app")
logger.debug("App is starting...")

testLogging:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
'''
Functions to set up the app logger
'''
import logging
import logging.config
import os

LOG_CONFIG = '../config/logging.conf'

def logpath(scriptdir, logfile):
    '''
    Return the path to the main log file; needed by the logging.yml
    use this for dynamic output log file paths & names
    '''
   log_file = os.path.join(scriptdir, logfile)
   print log_file
   print scriptdir
   print logfile
   return(logging.FileHandler(log_file))

def log_setup(config_file, logger_name):
    '''
    Set up the logger for the script
    config = path to YAML config file
    '''
# Config file relative to this file
    logging.config.fileConfig(config_file)
    return(logging.getLogger(logger_name))

logging.conf file:

[loggers]
keys=root

[handlers]
keys=consoleHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=DEBUG
handlers=consoleHandler
qualname=app

[logger_app]
level=DEBUG
handlers=consoleHandler
qualname=app
propagate=true

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)

[handler_fileHandler]
class=FileHandler
level=DEBUG
formatter=fileFormatter
args=('%(logfilename)s',)

[main]
()=__main__.logpath
level=DEBUG
formatter=simpleFormatter

[formatter_fileFormatter]
format=%(asctime)s (%(name)s:%(funcName)s:%(lineno)d:%(levelname)s) %
(message)s # %(module)s:
datefmt="%Y-%m-%d %H:%M:%S"

[formatter_simpleFormatter]
format=%(asctime)s (%(name)s:%(funcName)s:%(lineno)d:%(levelname)s) %(message)s # %(module)s:
datefmt="%Y-%m-%d %H:%M:%S"

user5359531 · Accepted Answer · 2017-07-24 18:48:01Z

2

I am doing a similar thing, here is how I have it set up

External config file in YAML format:

logging.yml

version: 1
formatters:
  default: # default debug logger
    format: '[%(asctime)s] (%(name)s:%(funcName)s:%(lineno)d:%(levelname)s) %(message)s' # %(module)s:
    datefmt: "%Y-%m-%d %H:%M:%S"
  info: # basic info logging, for easier reading
    format: '[%(levelname)-8s] %(message)s'
    datefmt: "%Y-%m-%d %H:%M:%S"
  console:
    format: '[%(asctime)s] (%(name)s:%(funcName)s:%(lineno)d:%(levelname)s) %(message)s'
    datefmt: "%Y-%m-%d %H:%M:%S"

handlers:
  console:
    class: logging.StreamHandler
    level: DEBUG
    formatter: console
    stream: ext://sys.stdout
  main:
    () : __main__.logpath # output file path
    level: DEBUG
    formatter: default

loggers:
  app:
    level: DEBUG
    handlers: [console, main]
    propagate: true
  parse:
    level: DEBUG
    handlers: [console, main]
    propagate: true
  tools:
    level: DEBUG
    handlers: [console, main]
    propagate: true
  data:
    level: DEBUG
    handlers: [console, main]
    propagate: true

Note the line () : __main__.logpath here, that calls a function called logpath in the main script in order to get the filehandler. I do this for conditional setting of the output file name. Put whatever filenaming or other Filehandler logic you need in there. See this in the 'main' app Python program:

app.py

!/usr/bin/env python
# -*- coding: utf-8 -*-

'''
My app
'''
# ~~~~~ LOGGING SETUP ~~~~~ #
# set up the first logger for the app
import os
import log as vlog
# path to the current script's dir
scriptdir = os.path.dirname(os.path.realpath(__file__))

def logpath():
    '''
    Return the path to the main log file; needed by the logging.yml
    use this for dynamic output log file paths & names
    '''
    global scriptdir
    return(vlog.logpath(scriptdir = scriptdir, logfile = 'log.txt'))

config_yaml = os.path.join(scriptdir,'logging.yml')
logger = vlog.log_setup(config_yaml = config_yaml, logger_name = "app")
logger.debug("App is starting...")

This is accompanied by log.py (imported as vlog in my main app);

log.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-
'''
Functions to set up the app logger
'''
import yaml
import logging
import logging.config
import os

def logpath(scriptdir, logfile = 'log.txt'):
    '''
    Return the path to the main log file; needed by the logging.yml
    use this for dynamic output log file paths & names
    '''
    log_file = os.path.join(scriptdir, logfile)
    return(logging.FileHandler(log_file))

def log_setup(config_yaml, logger_name):
    '''
    Set up the logger for the script
    config = path to YAML config file
    '''
    # Config file relative to this file
    loggingConf = open(config_yaml, 'r')
    logging.config.dictConfig(yaml.load(loggingConf))
    loggingConf.close()
    return(logging.getLogger(logger_name))

Additionally, any other modules of my own which I import in app.py (after setting up the logging there) includes this logging setup at the start of the module:

data.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-
'''
Module to do data stuff for my app
'''
import logging
logger = logging.getLogger("data")

I think this covers all the points you were talking about. It took me a while to figure this out myself. Hope it helps.

edited Jul 24, 2017 at 18:48

answered Jul 24, 2017 at 18:42

user5359531

3,6058 gold badges40 silver badges64 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Simply_me Over a year ago

This is perfect! Incredibly helpful. Thank you. I'll review this and let you know if anything comes up.

Simply_me Over a year ago

I got this to work for console output, but for some reason it doesnt create a log file.

user5359531 Over a year ago

Try hard-coding a path in logpath() at the line return(logging.FileHandler(log_file)), check the update I put in logging.yml where loggers: includes an entry (e.g. app) with handlers: [console, main] where main is the name of the log going to the file, and make sure that in your script you are calling a logger named under loggers there (e.g. app)

user5359531 Over a year ago

I ended up running into the same issue with messages not appearing, so I decided to start fresh and put together this bare-bones demo here

Simply_me Over a year ago

lucky me! It was exactly what I needed. Thank you again.

Collectives™ on Stack Overflow

Yet Another Python Logging Setup

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related