Customize key value in python structured (json) logging from config file

Question

I have to output my python job's logs as structured (json) format for our downstream datadog agent to pick them up. Crucially, I have requirements about what specific log fields are named, e.g. there must be a timestamp field which cannot be called e.g. asctime. So a desired log looks like:

{"timestamp": "2022-11-10 00:28:58,557", "name": "__main__", "level": "INFO", "message": "my message"}

I can get something very close to that with the following code:

import logging.config
from pythonjsonlogger import jsonlogger


logging.config.fileConfig("logging_config.ini", disable_existing_loggers=False)
logger = logging.getLogger(__name__)

logger.info("my message")

referencing the following logging_config.ini file:

[loggers]
keys = root

[handlers]
keys = consoleHandler

[formatters]
keys=json

[logger_root]
level=DEBUG
handlers=consoleHandler

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=json

[formatter_json]
class = pythonjsonlogger.jsonlogger.JsonFormatter
format=%(asctime)s %(name)s - %(levelname)s:%(message)s

...however, this doesn't allow flexibility about the keys in the outputted log json objects. e.g. my timestamp object is called "asctime" as below:

{"asctime": "2022-11-10 00:28:58,557", "name": "__main__", "levelname": "INFO", "message": "my message"}

I still want that asctime value (e.g. 2022-11-10 00:28:58,557), but need it to be referenced by a key called "timestamp" instead of "asctime". If at all possible I would strongly prefer a solution that adapts the logging.config.ini file (or potentially a yaml logging config file) with relatively minimal extra python code itself.

I also tried this alternative python json logging library which I thought provided very simple and elegant code, but unfortunately when I tried to use that, I didn't get my log statement to output at all...

Just use structlog.

wim
– wim

2022-11-10 06:45:27 +00:00
Commented Nov 10, 2022 at 6:45 — wim
– wim, Commented Nov 10, 2022 at 6:45

Vinay Sajip · Accepted Answer · 2022-11-10 11:39:57Z

3

You'll need to have a small, minimal amount of Python code, something like

# in mymodule.py, say

class CustomJsonFormatter(jsonlogger.JsonFormatter):
    def add_fields(self, log_record, record, message_dict):
        super(CustomJsonFormatter, self).add_fields(log_record, record, message_dict)
        log_record['timestamp'] = datetime.datetime.fromtimestamp(record.created).strftime('%Y-%m-%d %H:%M:%S') + f',{int(record.msecs)}'

and then change the configuration to reference it:

[formatter_json]
class = mymodule.CustomJsonFormatter
format=%(timestamp)s %(name)s - %(levelname)s:%(message)s

which would then output e.g.

{"timestamp": "2022-11-10 11:37:25,153", "name": "root", "levelname": "DEBUG", "message": "foo"}

answered Nov 10, 2022 at 11:39

Vinay Sajip

100k15 gold badges184 silver badges196 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Max Power Over a year ago

Hey thanks this looks perfect. Will accept soon after I get a chance to test it out.

piotrekkr Over a year ago

Thanks also. It is surprisingly hard to get proper ISO date in python logging with fractions of a second and time zone. Idk why it is not default format. No idea also why it is using local time zone without displaying it in logs. I used this log_record['timestamp'] = datetime.datetime.fromtimestamp(record.created, tz=datetime.UTC).isoformat() because I think record.created is actually in UTC and there is no need to format ISO time manually when there is isoformat().

Vinay Sajip Over a year ago

@piotrekkr The logging package was initially developed with Python 1.5.2, before datetime was added to Python (2.3). Although logging was also added in 2.3, there were users on older versions of Python who used logging, whose code had to be kept working.

piotrekkr Over a year ago

@VinaySajip We already have Python 3.13. Through all those years, was there no time to make it right? Or introduce new improved logging and deprecate old one? Do we need to use all those hacky solutions still? I just don't get it as not super advanced Python dev.

Vinay Sajip Over a year ago

Backward compatibility is taken very seriously. We don't want to break old code if we can help it. Feel free to propose new PRs that won't break existing tests and code.

Collectives™ on Stack Overflow

Customize key value in python structured (json) logging from config file

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related