How can I format this JSON String into a JSON object?

Question

I have the following JSON document being sent from an SNS HTTPs subscription to an API Gateway endpoint (backed by a Python2.7 Lambda function). In this case, I have a Cloudwatch Alarm that's configured to send to SNS, which then sends to this Gateway endpoint. SNS packages that alarm (which is JSON) into the "body" field of the message it sends to Gateway.

I need to extract the JSON document out of the "body" field, but by this point it's been properly mangled with escape characters and new lines and json.loads() is not liking it at all.

How can I read the value of "body" back into a JSON document using Python2.7 within a Lambda function? I've tried cleaning it by removing '\n' and '\', but I'm just striking out!

Here is the JSON as received by Lambda:

{
"body": "{\n  \"Type\" : \"Notification\",\n  \"MessageId\" : \"944c9xxx3-c98d636ff2c7\",\n  \"TopicArn\" : \"arn:aws:sns:us-west-2:xxx6xx:sxxxr-sns-topic\",\n  \"Subject\" : \"ALARM: \\\"hhh\\\" in US West (Oregon)\",\n  \"Message\" : \"{\\\"AlarmName\\\":\\\"hhh\\\",\\\"AlarmDescription\\\":null,\\\"AWSAccountId\\\":\\\"8xxx\\\",\\\"NewStateValue\\\":\\\"ALARM\\\",\\\"NewStateReason\\\":\\\"Threshold Crossed: 1 out of the last 1 datapoints [0.333370380661336 (13/06/18 18:06:00)] was greater than or equal to the threshold (0.1) (minimum 1 datapoint for OK -> ALARM transition).\\\",\\\"StateChangeTime\\\":\\\"2018-06-13T18:16:56.457+0000\\\",\\\"Region\\\":\\\"US West (Oregon)\\\",\\\"OldStateValue\\\":\\\"INSUFFICIENT_DATA\\\",\\\"Trigger\\\":{\\\"MetricName\\\":\\\"CPUUtilization\\\",\\\"Namespace\\\":\\\"AWS/EC2\\\",\\\"StatisticType\\\":\\\"Statistic\\\",\\\"Statistic\\\":\\\"AVERAGE\\\",\\\"Unit\\\":null,\\\"Dimensions\\\":[{\\\"name\\\":\\\"InstanceId\\\",\\\"value\\\":\\\"i-07bxxx26\\\"}],\\\"Period\\\":300,\\\"EvaluationPeriods\\\":1,\\\"ComparisonOperator\\\":\\\"GreaterThanOrEqualToThreshold\\\",\\\"Threshold\\\":0.1,\\\"TreatMissingData\\\":\\\"\\\",\\\"EvaluateLowSampleCountPercentile\\\":\\\"\\\"}}\",\n  \"Timestamp\" : \"2018-06-13T18:16:56.486Z\",\n  \"SignatureVersion\" : \"1\",\n  \"Signature\" : \"fFunXkjjxxxvF7Kmxxx\",\n  \"SigningCertURL\" : \"https://sns.us-west-2.amazonaws.com/SimpleNotificationService-xxx.pem\",\n  \"UnsubscribeURL\" : \"https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=axxxd\"\n}",
"resource": "/message",
"requestContext": {
    "requestTime": "13/Jun/2018:18:16:56 +0000",
    "protocol": "HTTP/1.1",
    "resourceId": "m4sxxxq",
    "apiId": "2v2cthhh",
    "resourcePath": "/message",
    "httpMethod": "POST",
    "requestId": "f41e8-8cbd-57ad9e625d12",
    "extendedRequestId": "xxx",
    "path": "/stage/message",
    "stage": "stage",
    "requestTimeEpoch": 1528913816627,
    "identity": {
        "userArn": null,
        "cognitoAuthenticationType": null,
        "accessKey": null,
        "caller": null,
        "userAgent": "Amazon Simple Notification Service Agent",
        "user": null,
        "cognitoIdentityPoolId": null,
        "cognitoIdentityId": null,
        "cognitoAuthenticationProvider": null,
        "sourceIp": "xxx",
        "accountId": null
    },
    "accountId": "xxx"
},
"queryStringParameters": {
    "id": "CBxxx69"
},
"httpMethod": "POST",
"pathParameters": null,
"headers": {
    "Content-Type": "text/plain; charset=UTF-8",
    "Via": "1.1 xxx.cloudfront.net (CloudFront)",
    "Accept-Encoding": "gzip,deflate",
    "CloudFront-Is-SmartTV-Viewer": "false",
    "x-amz-sns-subscription-arn": "arn:aws:sns:us-west-2:xxx:sxxx-nxxx-sns-topic:xxx",
    "CloudFront-Forwarded-Proto": "https",
    "X-Forwarded-For": "54.240.xxx, 54.182.xxx",
    "CloudFront-Viewer-Country": "US",
    "User-Agent": "Amazon Simple Notification Service Agent",
    "X-Amzn-Trace-Id": "Root=1-5b21xxx53acea6642317ed4",
    "x-amz-sns-topic-arn": "arn:aws:sns:us-west-2:xxxx:sxxxier-sns-topic",
    "Host": "2vxxx.execute-api.us-west-2.amazonaws.com",
    "X-Forwarded-Proto": "https",
    "X-Amz-Cf-Id": "xxx",
    "CloudFront-Is-Tablet-Viewer": "false",
    "X-Forwarded-Port": "443",
    "x-amz-sns-message-type": "Notification",
    "CloudFront-Is-Mobile-Viewer": "false",
    "x-amz-sns-message-id": "xxx",
    "CloudFront-Is-Desktop-Viewer": "true"
},
"stageVariables": null,
"path": "/message",
"isBase64Encoded": false
}

You can't just keep removing the garbage until it's clean? Like: body.replace('\"', '').replace('\n', '')...... — sehafoc
– sehafoc, Commented Jun 13, 2018 at 21:25
Check out this one, I think you need to convert the string first , then you an do normal json decoding: stackoverflow.com/questions/24242433/… (maybe this answer is best: stackoverflow.com/a/24242596/399696). There's a python 3 version as well. — dmgig
– dmgig, Commented Jun 13, 2018 at 21:27
@sehafoc you should never try to modify JSON-in-JSON with string replacement manipulation. That isn't garbage. — Michael - sqlbot
– Michael - sqlbot, Commented Jun 13, 2018 at 23:36
"event – AWS Lambda uses this parameter to pass in event data to the handler. This parameter is usually of the Python dict type". You said below that the output in your question is from json.dumps(event), isn't event already a Python dict? — G_M
– G_M, Commented Jun 14, 2018 at 4:35
@Michael-sqlbot That's sound advice, The body JSON section looks like it's be grabbed as an output of a pretty printer or something, which would violate JSON rules. Though in my tests everything json.loads() cleanly anyways. Which lead me to believe there is something else there. — sehafoc
– sehafoc, Commented Jun 15, 2018 at 21:00

powo · Accepted Answer · 2018-06-13 21:28:02Z

1

if i use your pasted sample as a raw string, it works well:

>>> j = r'''...your sample pasted here...'''
>>> data = json.loads(j)
>>> bodydata = json.loads(data['body'])
>>> bodydata['Type']
u'Notification'

seems, that what you pasted above is the repr form, printed out with Python

answered Jun 13, 2018 at 21:28

powo

5301 gold badge7 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user212869 Over a year ago

Yeah, what I pasted was the json.dumps(event) output. Printing out event directly ports it out as unicode. Whenever I execute what you have, I get ValueError: Expecting : delimiter: line 42 column 27 (char 2808). How would you suggest I convert the input variable to a raw string in the script? Thanks!

powo Over a year ago

Since event is already a python dict, all you may need is json.loads(event['body'])

Collectives™ on Stack Overflow

How can I format this JSON String into a JSON object?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related