Store python dictionary in JSON file dynamically

Question

I have a python dictionary with keys in the format Aa123 and each value is a list. Now I need to store this data permanently in some format or in any database so that I can retrieve it easily. I thought JSON would be better for this and tried to store all the data in JSON format and then some application could use this file. My JSON format should be like,

[
  {  "firstLetter" : "A",
     "remaining" : [
        {
           "secondLetter" : "a",
           "ID" : [
              {"id" : "Aa123", "listOfItems" : ["ABC123","ASD100"]},
              {"id" : "Aa100", "listOfItems" : ["ABC123","COD101"]}
           ]
        },
        {
           "secondLetter" : "b",
           "ID" : [
              {"id" : "Ab100", "listOfItems" : ["ABC123","ASD100"]}
           ]
        }
     ]
  },
  {  "firstLetter" : "B",
     "remaining" : [
        {
           "secondLetter" : "a",
           "ID" : [                  
              {"id" : "Ba106", "listOfItems" : ["AUD123","CML101"]}
           ]
        },
        {
           "secondLetter" : "b",
           "ID" : [
              {"id" : "Bb153", "listOfItems" : ["AER113","ASD100"]},
              {"id" : "Bb100", "listOfItems" : ["ATC123","ASD500"]}
           ]
        }
     ]
  }
]

I know how to dump python dictionary into JSON, but that doesn't provide easy data query. My question is "how to store this python dictionary(that I obtained by running a python program)in required format(like the one shown above) that makes data query easy. Thanks!

Check for NoSQL databases like MongoDB. That is what you need — Moinuddin Quadri
– Moinuddin Quadri, Commented Nov 2, 2016 at 10:12
Is the problem storing the data, or is it converting the data to the format shown in your question? Also, how will you be querying the data? Are you trying to implement a trie/prefix tree? — mhawke
– mhawke, Commented Nov 2, 2016 at 11:08

chthonicdaemon · Accepted Answer · 2016-11-02 11:25:25Z

It sounds like you may benefit from tinydb. It stores values directly in a JSON file and provides methods for querying.

So to store your values, we do

from tinydb import TinyDB, Query

db = TinyDB('test.json')
values = [{'firstLetter': 'A',
  'remaining': [{'ID': [{'id': 'Aa123', 'listOfItems': ['ABC123', 'ASD100']},
     {'id': 'Aa100', 'listOfItems': ['ABC123', 'COD101']}],
    'secondLetter': 'a'},
   {'ID': [{'id': 'Ab100', 'listOfItems': ['ABC123', 'ASD100']}],
    'secondLetter': 'b'}]},
 {'firstLetter': 'B',
  'remaining': [{'ID': [{'id': 'Ba106', 'listOfItems': ['AUD123', 'CML101']}],
    'secondLetter': 'a'},
   {'ID': [{'id': 'Bb153', 'listOfItems': ['AER113', 'ASD100']},
     {'id': 'Bb100', 'listOfItems': ['ATC123', 'ASD500']}],
    'secondLetter': 'b'}]}]
for value in values:
    db.insert(value)

To query, we do

>>> Q = Query()
>>> db.search(Q.firstLetter == "A")

[{'firstLetter': 'A',
  'remaining': [{'ID': [{'id': 'Aa123', 'listOfItems': ['ABC123', 'ASD100']},
     {'id': 'Aa100', 'listOfItems': ['ABC123', 'COD101']}],
    'secondLetter': 'a'},
   {'ID': [{'id': 'Ab100', 'listOfItems': ['ABC123', 'ASD100']}],
    'secondLetter': 'b'}]}]

Vlad B · Accepted Answer · 2016-11-02 14:09:13Z

1

Other than a NoSQL database like MongoDB and Cassandra, if you are using PostgreSQL, have a look at the hstore data type, specifically psycopg2's hstore functionality. This may be of help if you want to be able to query your data more easily.

edited Nov 2, 2016 at 14:09

answered Nov 2, 2016 at 10:37

Vlad B

3333 silver badges11 bronze badges

Comments

Ivan Chaer · Accepted Answer · 2016-11-02 19:01:30Z

In the example below, I output the data to a JSON, and also store the dict using Shelves.

import json
import codecs
import shelve

formatted_item = {
    "data": [
      {  "firstLetter" : "A",
         "remaining" : [
            {
               "secondLetter" : "a",
               "ID" : [
                  {"id" : "Aa123", "listOfItems" : ["ABC123","ASD100"]},
                  {"id" : "Aa100", "listOfItems" : ["ABC123","COD101"]}
               ]
            },
            {
               "secondLetter" : "b",
               "ID" : [
                  {"id" : "Ab100", "listOfItems" : ["ABC123","ASD100"]}
               ]
            }
         ]
      },
      {  "firstLetter" : "B",
         "remaining" : [
            {
               "secondLetter" : "a",
               "ID" : [                  
                  {"id" : "Ba106", "listOfItems" : ["AUD123","CML101"]}
               ]
            },
            {
               "secondLetter" : "b",
               "ID" : [
                  {"id" : "Bb153", "listOfItems" : ["AER113","ASD100"]},
                  {"id" : "Bb100", "listOfItems" : ["ATC123","ASD500"]}
               ]
            }
         ]
      }
    ]
}


###
# JSON

# *** Store

output = json.dumps(dict(formatted_item), sort_keys=True, indent=4, separators=(',', ': '))
json_file_path = 'temp.json'
with codecs.open(json_file_path, 'wb', encoding='utf8') as file:
    file.write(output)

# *** Read

with open(json_file_path) as json_file:
    json_data = json.load(json_file)

# *** Print

print json_data
print json_data['data'][0]['firstLetter']


###
# Shelves

# *** Store

shelf_file_path = 'temp.log'
shelf = shelve.open(shelf_file_path)
shelf.update(formatted_item)
shelf.close()

# *** Read

loaded_shelf = shelve.open(shelf_file_path)

# *** Print

print loaded_shelf
print loaded_shelf['data'][0]['firstLetter']

Neill · Accepted Answer · 2016-11-02 10:20:50Z

0

You can save Python Dict objects directly on MongoDB

For more information, refer to this: http://api.mongodb.com/python/current/tutorial.html

answered Nov 2, 2016 at 10:20

Neill

5171 gold badge6 silver badges16 bronze badges

Comments

mootmoot · Accepted Answer · 2016-11-02 11:19:58Z

-1

DBMS doesn't deal with JSON query out of the box very well.

You can now store JSON file in supported RDBMS, NoSQL for quick query. Nevertheless, DBMS that support JSON data type, try workaround by skimming the json structure and index them massively. NoSQL is no better than RDBMS when dealing with json data type.

So you must always read particular RDBMS/NoSQL documentation before storing json file, especially when you have tons of json data.

answered Nov 2, 2016 at 11:19

mootmoot

13.2k5 gold badges49 silver badges44 bronze badges

Collectives™ on Stack Overflow

Store python dictionary in JSON file dynamically

5 Answers 5

Comments

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related