Fix too many logs using json config file in python

Question

I've a python application which is getting bigger day a day and now I'm trying to change its logging system. Thus, I've prepared a json config file for logging properties. Like this:

{
    "version": 1,
    "disable_existing_loggers": false,
    "formatters": {
        "simple": {
            "format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s",
            "datefmt": "%Y-%m-%d %H:%M:%S"
        }
    },

    "handlers": {
        "rotate_timed": {
            "level": "DEBUG",
            "class": "logging.handlers.TimedRotatingFileHandler",
            "formatter": "simple",
            "when": "midnight",
            "backupCount": 5,
            "filename": "/var/log/v4m_agentd.log",
            "encoding": "utf8"
        }
    },

    "loggers": {
        "my_module": {
            "level": "DEBUG",
            "handlers": ["rotate_timed"]
        }
    },

    "root": {
        "level": "DEBUG",
        "handlers": ["rotate_timed"],
        "propagate": false
    }
}

My main program get logger properties using:

logging_config_json_file = open(os.path.join(os.path.dirname(os.path.realpath(__file__)), "logging.json"))
parsed_logging_data = json.load(logging_config_json_file)
logging.config.dictConfig(parsed_logging_data)

My modules call the logger like:

class StoreData(object):
    def __init__(self):
        self.logger = logging.getLogger(__name__)

Since I've changed logging method, to log using a config file, I've a lot of log lines in my .log file, due to all imported modules in my classes (like: apscheduler, werkzeug, ...) are using this config file to log in their messages in my log file and this doesn't happend before.

Before:

2015-04-09 10:29:18 - Agent Get Data - DEBUG - GETTING DATA
2015-04-09 10:29:18 - Prepare DB Data - DEBUG - -----Init prepare data for DB-----
2015-04-09 10:29:21 - Jobs Listener - INFO - The job 'Get Data' worked :)       Next run at: 2015-04-09 10:29:23

After:

2015-04-10 10:27:59 - Agent Get Data - DEBUG - GETTING DATA
2015-04-10 10:27:59 - Prepare DB Data - DEBUG - -----Init prepare data for DB-----
2015-04-10 10:28:02 - apscheduler.executors.default - INFO - Job "Get Data (trigger: interval[0:00:05], next run at: 2015-04-10 10:28:04 CEST)" executed successfully
2015-04-10 10:28:02 - Jobs Listener - INFO - The job 'Get Data' worked :)       Next run at: 2015-04-10 10:28:04
2015-04-10 10:28:04 - apscheduler.scheduler - DEBUG - Looking for jobs to run
2015-04-10 10:28:04 - apscheduler.scheduler - DEBUG - Next wakeup is due at 2015-04-10 10:28:09.847414+02:00 (in 4.998605 seconds)
2015-04-10 10:28:04 - apscheduler.executors.default - INFO - Running job "Get Data (trigger: interval[0:00:05], next run at: 2015-04-10 10:28:09 CEST)" (scheduled at 2015-04-10 10:28:04.847414+02:00)

I only want to write my own log messages. How could I fix it?

Thanks a lot!

bruno desthuilliers · Accepted Answer · 2018-07-31 09:16:46Z

1

First (slightly unrelated but anyway...) you have an error in your config: your root logger is defined at the top-level instead of being under the loggers key.

Then you may want to set the root logger's level way higher (WARNING or ERROR - WARNING being the default and a sensible one) so you only get the most important messages from other lib's loggers.

Once done with it, you can also use two distinct handlers - one for the root logger and a dedicated one for your own libs - so you get one log with important messages from all libs (including yours) and another log with all messages from your own libs and nothing from the others.

answered Jul 31, 2018 at 9:16

bruno desthuilliers

78.3k6 gold badges103 silver badges129 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

dpatino Over a year ago

Perfect, the truth is that I had not noticed. This makes it better and more modular for the different sources of logs

David · Accepted Answer · 2018-07-31 08:37:04Z

0

You have to set disable_existing_loggers to true in your logging.json.

edited Jul 31, 2018 at 8:37

David

1,2105 gold badges14 silver badges31 bronze badges

answered Jul 31, 2018 at 6:50

Paolo Venturi

1

Collectives™ on Stack Overflow

Fix too many logs using json config file in python

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related