datadog log processing: how to unescape nested json

Question

I'm sending json formatted logs messages to datadog:

  "Headers": {
    "Content-Type": [
      "application/json"
    ]
  },
  "Method": "POST",
  "Body": "{\"test\": 1234, \"test2\": {\"test3\": 1245}}",
  "URL": "https://test",
  "Host": "test",
  "UserAgent": ""
}

The issue is with the field Body. I want datadog to pick this up as json so it's parsed. So I'm looking into log pipelines and in particular the grok processor to preprocess the log message and unescape the Body field.

%{data::json} as a parsing rule works but doesn't unescape Body. I tried stuff like: %{data::json{source => "Body" target => "Body"}} but datadog marks the rule as invalid.

Is there a better way to achieve the unescaping in datadog?
How can I create a valid grok rule for datadog that unescapes the Body field?

DaniWestSide · Accepted Answer · 2024-06-05 12:46:49Z

I know this is an old question, but ran into the same issue and my team found a nice solution to it.

We ran into this problem when using Datadog integration with Azure Container Apps, where the application logs are sent inside "properties.Log" field, and characters are escaped, for example:

{
(...)
"properties": {
(...)
"Log": "{\"level\":\"info\",\"time\":1717269956664,\"msg\":\"XXXXX\"}",
}
(...)
}

By setting a simple grok processor in Datadog like the following, targeted at the field where the nested JSON that we want to unescape is, i.e. Advanced Settings -> Extract From: "properties.Log" in this case, solved the issue:

rule %{data::json}

This solution parses and extracts the fields within the nested JSON, into to the root of the JSON document. It works despite the fact that this grok parser provides an empty output in the preview section, when the sample escaped input JSON is provided, which would lead us to believe it won't work.

Collectives™ on Stack Overflow

datadog log processing: how to unescape nested json

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related