process multiple jason arrays from a .dat file in python

Question

I am new to json data processing and stuck with this issue. Data in my input file looks like this -

[{"key1":"value1"},{"key2":"value2"}] [{"key3":"value3"},{"key4":"value4"}]

I tried to read using

json.load(file)

or by

with open(file) as f:
 json.loads(f)

tried with pandas.read_json(file, orient="records") as well

each of these attempts failed with stating Extra data: line 1 column n (char n) issue

Can someone guide how best to parse this file? I am not in favor writing a manual parser which may fail to scale later

P.S. There is no , between two arrays TIA

You seem to have named a variable file which does not make it a file from which you can read with open. This is still a text variable. — sophros
– sophros, Commented Feb 19, 2018 at 13:40
json.load will take file pointer input, and json.loads will take a string as input. — Hara
– Hara, Commented Feb 19, 2018 at 13:54

Hara · Accepted Answer · 2018-02-19 17:16:10Z

1

Your Json file content has issue.

1. If , between arrays:

Code:

import json

with open("my.json") as fp:
    data = json.load(fp)  # data = json.loads(fp.read())

print data

your file content can be eithor of these.

Option1: Use outer most square bracket for your json content.

[[ {"key1":"value1"}, {"key2":"value2"}], [{"key3":"value3"}, {"key4":"value4"}]]

Option2: use only one square bracket.

[ {"key1":"value1"}, {"key2":"value2"}, {"key3":"value3"}, {"key4":"value4"}]

2. If no , between arrays:

code: Just writing as per the given JSON format.

def valid_json_creator(given):
    replaced = given.replace("}] [{", "}],[{")
    return "[" + replaced + "]"


def read_json():
    with open("data.txt") as fp:
        data = fp.read()
    valid_json = valid_json_creator(data)
    jobj = json.loads(valid_json)
    print(jobj)


if __name__ == '__main__':
    read_json()

This code works for JSON if it is in the following format.

Note no , between arrays, but space is there.

[{"key0":"value0"},{"key1":"value41"}] [{"key1":"value1"},{"key2":"value42"}] [{"key2":"value2"},{"key3":"value43"}] [{"key3":"value3"},{"key4":"value44"}] [{"key4":"value4"},{"key5":"value45"}] [{"key5":"value5"},{"key6":"value46"}] [{"key6":"value6"},{"key7":"value47"}] [{"key7":"value7"},{"key8":"value48"}] [{"key8":"value8"},{"key9":"value49"}] [{"key9":"value9"},{"key10":"value410"}] [{"key10":"value10"},{"key11":"value411"}] [{"key11":"value11"},{"key12":"value412"}] [{"key12":"value12"},{"key13":"value413"}] [{"key13":"value13"},{"key14":"value414"}] [{"key14":"value14"},{"key15":"value415"}] [{"key15":"value15"},{"key16":"value416"}] [{"key16":"value16"},{"key17":"value417"}] [{"key17":"value17"},{"key18":"value418"}] [{"key18":"value18"},{"key19":"value419"}] [{"key19":"value19"},{"key20":"value420"}]

edited Feb 19, 2018 at 17:16

answered Feb 19, 2018 at 13:48

Hara

1,5024 gold badges19 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

CARTman Over a year ago

please note there is no , between arrays

Hara Over a year ago

Ohh..! that's again another good point. :) thanks for pointing me.

Hara Over a year ago

@CARTman: do you want the file to be in the same format without a comma in between arrays?

sophros · Accepted Answer · 2018-02-19 13:39:42Z

0

What you have tested is reading from a structure that corresponds to the JSON file (which by definition is text, not Python data structure).

Test:

file = '[{"key1":"value1"},{"key2":"value2"}],[{"key3":"value3"},{"key4":"value4"}]'

This should work better. But wait... you do not seem to provide a list or dict at the top level of your to-be JSON! Hence the error:

ValueError: Extra data: line 1 column 38 - line 1 column 76 (char 37 - 75)

Change it then to (note the additional list opening and closing brackets at the beginning and end):

file = '[[{"key1":"value1"},{"key2":"value2"}],[{"key3":"value3"},{"key4":"value4"}]]'

This will work with:

json.load(file)

but not with:

with open(file) as f:
 json.loads(f)

as your text variable is not a file! You would want to store the contents of the variable named file to a file and pass the path to that file:

with open(r'C:\temp\myfile.json') as f:
 json.loads(f)

For the code to work properly.

answered Feb 19, 2018 at 13:39

sophros

17.3k12 gold badges52 silver badges84 bronze badges

2 Comments

CARTman Over a year ago

I edited my file to put [ ] as outermost bracket, still getting this error Expecting value: line 1 column 1 (char 0)

CARTman Over a year ago

Please note there is no , between arrays

Collectives™ on Stack Overflow

process multiple jason arrays from a .dat file in python

2 Answers 2

3 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related