Issue parsing multiline JSON file using Python

Question

I am trying to parse a JSON multiline file using json library in Python 2.7. A simplified sample file is given below:

{
"observations": {
    "notice": [
        {
            "copyright": "Copyright Commonwealth of Australia 2015, Bureau of Meteorology. For more information see: http://www.bom.gov.au/other/copyright.shtml http://www.bom.gov.au/other/disclaimer.shtml",
            "copyright_url": "http://www.bom.gov.au/other/copyright.shtml",
            "disclaimer_url": "http://www.bom.gov.au/other/disclaimer.shtml",
            "feedback_url": "http://www.bom.gov.au/other/feedback"
        }
    ]
}
}

My code is as follows:

import json

with open('test.json', 'r') as jsonFile:
    for jf in jsonFile:
        jf = jf.replace('\n', '')
        jf = jf.strip()
        weatherData = json.loads(jf)
        print weatherData

Nevertheless, I get an error as shown below:

Traceback (most recent call last):
File "test.py", line 8, in <module>
weatherData = json.loads(jf)
File "/home/usr/anaconda2/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/home/usr/anaconda2/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/home/usr/anaconda2/lib/python2.7/json/decoder.py", line 380, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting object: line 1 column 1 (char 0)

Just to do some testing, I modified the code such that after removing newlines and striping away the leading and trailing white spaces, I write the contents to another file (with the json extension). Surprisingly, when I read back the latter file, I do not get any error and the parsing is successful. The modified code is as follows:

import json

filewrite = open('out.json', 'w+')

with open('test.json', 'r') as jsonFile:
    for jf in jsonFile:
        jf = jf.replace('\n', '')
        jf = jf.strip()
        filewrite.write(jf)

filewrite.close()

with open('out.json', 'r') as newJsonFile:
    for line in newJsonFile:
        weatherData = json.loads(line)
        print weatherData

The output is as follows:

{u'observations': {u'notice': [{u'copyright_url': u'http://www.bom.gov.au/other/copyright.shtml', u'disclaimer_url': u'http://www.bom.gov.au/other/disclaimer.shtml', u'copyright': u'Copyright Commonwealth of Australia 2015, Bureau of Meteorology. For more information see: http://www.bom.gov.au/other/copyright.shtml http://www.bom.gov.au/other/disclaimer.shtml', u'feedback_url': u'http://www.bom.gov.au/other/feedback'}]}}

Any idea what might be going on when new lines and white spaces are stripped before using json library?

Okezie · Accepted Answer · 2015-12-25 13:42:35Z

10

You will go crazy if you try to parse a json file line by line. The json module has helper methods to read file objects directly or strings i.e. the load and loads methods. load takes a file object (as shown below) for a file that contains json data, while loads takes a string that contains json data.

Option 1: - Preferred

import json
with open('test.json', 'r') as jf:
    weatherData = json.load(jf)
    print weatherData

Option 2:

import json
with open('test.json', 'r') as jf:
    weatherData = json.loads(jf.read())
    print weatherData

If you are looking for higher performance json parsing check out ujson

answered Dec 25, 2015 at 13:42

Okezie

5,1383 gold badges28 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

hypersonics Over a year ago

Thank you @OkezieE. Loading the entire file via load does the trick.

fodma1 · Accepted Answer · 2015-12-25 10:26:20Z

6

In the first snippet, you try to parse it line by line. You should parse it all at once. The easiest is to use json.load(jsonfile). (The jf variable name is misleading as it's a string). So the correct way to parse it:

import json

with open('test.json', 'r') as jsonFile:
    weatherData = json.loads(jsonFile)

Although it's a good idea to store the json in one line, as it's more concise.

In the second snippet your problem is that you print it as unicode string which is and u'string here' is python specific. A valid json uses double quotation marks

answered Dec 25, 2015 at 10:26

fodma1

3,5451 gold badge33 silver badges52 bronze badges

2 Comments

hypersonics Over a year ago

I tried your approach of loading the entire file and parsing all at once, but It fails. By the way, I have tried parsing json files earlier line by line without having any issues, expect that each dictionary never exceeded about a 100 characters long.

fodma1 Over a year ago

For my it was working fine with the json you provided. Try using the Sublime JSON plugin for converting the json back and forth: github.com/dzhibas/SublimePrettyJson It also validates the json, so you can investigate the issue further. For more detailed linting try jsonlint.com

Vlad M · Accepted Answer · 2015-12-25 10:52:57Z

2

FYI, you can have both files opened in single with statement:

with open('file_A') as in_, open('file_B', 'w+') as out_:
    # logic here
    ...

answered Dec 25, 2015 at 10:52

Vlad M

4773 silver badges10 bronze badges

3 Comments

Hamid Pourjam Over a year ago

While this code may answer the question, it is better to explain what it does and add some references to it.

President James K. Polk Over a year ago

This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - From Review

Vlad M Over a year ago

@James: agreed. at the moment of answering I was not able to comment on questions and thanks to upvote now I can :-)

Collectives™ on Stack Overflow

Issue parsing multiline JSON file using Python

3 Answers 3

1 Comment

2 Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

2 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related