Read JSON array from a text file in python

Question

I am trying to read a text file with a JSON array in it. Can someone help me to find my mistake?

   ["io", {"in": 8, "out": 0, "dev": "68", "time": 1532035082.614868}]
   ["io", {"in": 0, "out": 0, "dev": "68", "time": 1532035082.97122}]
   ["test", {"A": [{"para1":[], "para2": true, "para3": 68, "name":"", "observation":[[2,3],[3,2]],"time": 1532035082.97122}]}]

I did not manage to have it run with

  import gzip
  with gzip.open('myfile', 'rb') as f
      json_data = jsonload(f)
  print(json_data)

I have an error: json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 278) I can see that my file is not a JSON file but represents a JSON array I manage to have it work with pandas but I'd like to find out how to do it without pandas.

import pandas as pd
data = pd.read_json('myfile', lines=True)
print(data.head())

Arty · Accepted Answer · 2021-03-24 11:47:31Z

2

data = [json.loads(e) for e in f if e.strip()]

Example of usage

answered Mar 24, 2021 at 11:47

Arty

16.9k6 gold badges60 silver badges96 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Robson · Accepted Answer · 2021-03-24 11:43:29Z

0

The data in your text file is not a valid JSON. These are actually three JSON files one after another.

You have at last two options for fixing it:

put all the JSON to a single one, for example by making a list at the top of the structure
read your file line by line and load JSONs separately

answered Mar 24, 2021 at 11:43

Robson

8135 gold badges23 silver badges44 bronze badges

3 Comments

Stephen Over a year ago

So what I shall do?

Robson Over a year ago

I've updated my answer with two suggestions.

Stephen Over a year ago

Hi @Robson, I did option 1. with open.('myfile', 'rb') as f: lines=f.readlines()

Stephen · Accepted Answer · 2021-04-20 09:07:10Z

0

I found 2 options:

option1: pandas

pd.read_json(filepath,compression='infer', orient='records, lines=True)

option 2: read line by line

with gzip.open('myfile', 'rb') as f:
     lines=f.readlines()
     data = [json.loads(l) for l in lines]
print(data)

Option 1 is fastest:

option 1 = 0.31 s

option 2 = 0.42 s

edited Apr 20, 2021 at 9:07

answered Mar 24, 2021 at 12:57

Stephen

3193 silver badges9 bronze badges

Collectives™ on Stack Overflow

Read JSON array from a text file in python

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related