How to change structure of json data and add new fields on python

Question

I am trying to the structure of my json data, but I cant get my head around it. This is the sample of the data that I have

[<OldSample {u'counter_name': u'cpu_util', u'user_id': u'7bffa12f482840c7801e3e01e160c8cb', u'resource_id': u'ef392c3d-74fa-43fe-87c5-7e117b6d8a09', u'timestamp': u'2015-07-01T15:13:55', u'counter_volume': 0.034999999999999996, u'resource_metadata': {u'ramdisk_id': u'None', u'flavor.vcpus': u'1', u'OS-EXT-AZ.availability_zone': u'nova', u'display_name': u'ubuntu-io', u'flavor.id': u'596642d8-0813-4ae9-aec4-0105fdf05761', u'status': u'active', u'ephemeral_gb': u'0', u'flavor.name': u'm1.small.io', u'disk_gb': u'20', u'kernel_id': u'None', u'image.id': u'5776a360-0953-4c93-931d-a6e3616fb8dc', u'flavor.ram': u'2048', u'host': u'9fa544f5c47569db21d50bc6c0765296316a56bd6baf6b04d705686a', u'flavor.ephemeral': u'0', u'image.name': u'ubuntu-io', u'image_ref_url': u'link': u"[{'href': 'link', 'rel': 'bookmark'}]", u'cpu_number': u'1', u'flavor.disk': u'20', u'root_gb': u'20', u'name': u'instance-000000d3', u'memory_mb': u'2048', u'instance_type': u'596642d8-0813-4ae9-aec4-0105fdf05761', u'vcpus': u'1', u'image_ref': u'5776a360-0953-4c93-931d-a6e3616fb8dc', u'flavor.links': u"[{'href': 'link', 'rel': 'bookmark'}]"}, u'source': u'openstack', u'counter_unit': u'%', u'recorded_at': u'2015-07-01T15:13:56.006000', u'project_id': u'1670f0e56fb6421cb83d81b60b149c04', u'message_id': u'ca8ea466-2003-11e5-a764-002590e64886', u'counter_type': u'gauge'}>, <OldSample {u'counter_name': u'cpu_util', u'user_id': u'7bffa12f482840c7801e3e01e160c8cb', u'resource_id': u'0c8b6b26-3340-41e3-ac8b-cc38f15d3570', u'timestamp': u'2015-07-01T15:08:32', u'counter_volume': 5.4399999999999995, u'resource_metadata': {u'ramdisk_id': u'None', u'flavor.vcpus': u'1', u'OS-EXT-AZ.availability_zone': u'nova', u'display_name': u'kalman_instance', u'flavor.id': u'1', u'status': u'active', u'ephemeral_gb': u'0', u'flavor.name': u'm1.tiny', u'disk_gb': u'1', u'kernel_id': u'None', u'image.id': u'1c9b08f0-d1fa-4acc-a11c-87b77310158c', u'flavor.ram': u'512', u'host': u'25aa71ded460ea9d4bf52e1aac34017691699cb5e4e389704d738bed', u'flavor.ephemeral': u'0', u'image.name': u'cirros', u'image_ref_url': u'http://192.168.26.1:8774/d1d65b6feab741a6a2905e6197cb15ee/images/1c9b08f0-d1fa-4acc-a11c-87b77310158c', u'image.links': u"[{'href': 'http://192.168.26.1:8774/d1d65b6feab741a6a2905e6197cb15ee/images/1c9b08f0-d1fa-4acc-a11c-87b77310158c', 'rel': 'bookmark'}]", u'cpu_number': u'1', u'flavor.disk': u'1', u'root_gb': u'1', u'name': u'instance-0000013c', u'memory_mb': u'512', u'instance_type': u'1', u'vcpus': u'1', u'image_ref': u'1c9b08f0-d1fa-4acc-a11c-87b77310158c', u'flavor.links': u"[{'href': 'http://192.168.26.1:8774/d1d65b6feab741a6a2905e6197cb15ee/flavors/1', 'rel': 'bookmark'}]"}, u'source': u'openstack', u'counter_unit': u'%', u'recorded_at': u'2015-07-01T15:08:32.459000', u'project_id': u'1670f0e56fb6421cb83d81b60b149c04', u'message_id': u'09b5173e-2003-11e5-ac7a-002590e64b12', u'counter_type': u'gauge'}>]

Which I want to change in something like this

 message: {
      columns: [
        ["y": 0.043, "x": "2015-06-30T15:53:55"],
        ["y": 0.045, "x": "2015-06-30T15:53:55"]
      ]

Here is my code,

def clean_Json(data):
  for each in data:
    timestamp = each.timestamp
    volume =  each.counter_volume
    i = json.dumps({'x': timestamp, 'y': volume})
    print i

clean_Json(data)

Result:

{"y": 4.101666666666667, "x": "2015-04-10T15:18:18"}
{"y": 5.471666666666666, "x": "2015-04-10T14:48:18"}

The problem is that

1. there is no comma at the end of each line. 2. when I try to add Square brackets I get syntax error.

I cant seem to move forward from json.dumps({'x': timestamp, 'y': volume})

Structure you want to achieve isn't a JSON (JSON objects use {} brackets, not []), is this really what you want? — Tupteq
– Tupteq, Commented Jul 1, 2015 at 11:19
@Tupteq I am afraid so, actually what I want to achieve with this is something that I can plug in EON charts in pubnub pubnub.com/developers/eon/chart/spline — Imo
– Imo, Commented Jul 1, 2015 at 11:30
So, you probably shouldn't use JSON library, because you don't need a JSON. Consider creating your own formatting using idioms like ','.join(items). — Tupteq
– Tupteq, Commented Jul 1, 2015 at 11:35

Tupteq · Accepted Answer · 2015-07-02 14:53:32Z

1

How about this?

result_rows = []
for row in data:
    # Format output as desired
    formatted = """["y": %0.3f, "x": "%s"]""" % (row.counter_volume, row.timestamp)

    #Append to result list
    result_rows.append(formatted)

# Pythonic way to join strings putting something (comma+newline) between items
print(',\n'.join(result_rows))

edited Jul 2, 2015 at 14:53

answered Jul 1, 2015 at 13:29

Tupteq

3,1141 gold badge23 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

11 Comments

Imo Over a year ago

Thanks a lot, but here is another error for row in data.split("\n"): AttributeError: 'list' object has no attribute 'split'--- this is indeed a pain

Tupteq Over a year ago

If your data is already list (not string), then you don't need to split it, just use it directly for row in data:.

Imo Over a year ago

Now It raises an attribute error, File "data.py", line 57, in <module> row = row.strip() raise AttributeError(k) AttributeError: strip

Tupteq Over a year ago

It looks like you input data isn't string, of so, what exactly it is? Can you provide some real data, because content you provided in question converts without any problem.

Imo Over a year ago

provided complete data above

|

Ananth · Accepted Answer · 2015-07-01 11:28:23Z

0

In JSON, [] represent arrays, and {} represent objects. It is not possible to have an array with key-value pairs, like you have in your desired output for individual objects in the columns array. They are objects so they must be enclosed in {} only. You must implement your own string concatenation logic if you want square brackets for sure.

def get_clean_Json(data):
  return [{ "x": each['timestamp'], "y": each['counter_volume'] } for each in data]

Testing above code:

st = '[{"timestamp": "2015-06-30T15:53:55", "counter_volume": 0.043}, {"timestamp": "121", "counter_volume": 0.045}]'
data = json.loads(st)
j = get_clean_Json(data)

json.dumps({ 'message': { 'columns': j } })

Output:

{"message": {"columns": [{"y": 0.043, "x": "2015-06-30T15:53:55"}, {"y": 0.045, "x": "121"}]}}

edited Jul 1, 2015 at 11:28

answered Jul 1, 2015 at 11:17

Ananth

4,4252 gold badges23 silver badges26 bronze badges

1 Comment

Imo Over a year ago

There is a stickler in the data [<OldSample, (see data example above first line) and when I try your code above it gives me the error TypeError: 'OldSample' object has no attribute 'getitem'

Imo · Accepted Answer · 2015-07-06 11:57:53Z

0

This is how I solved the above problem:

thing = {}
msg = {}
cols = []


for row in data:
    col = {}
    col = {"x": row.timestamp, "y": row.counter_volume}
    cols.append(col)

msg['columns'] = cols

thing['message'] = msg

print json.dumps(thing, indent=4)

answered Jul 6, 2015 at 11:57

Imo

1,4754 gold badges30 silver badges58 bronze badges

Collectives™ on Stack Overflow

How to change structure of json data and add new fields on python

3 Answers 3

11 Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

11 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related