Python 3 : Storing Data without loading it into memory

Question

I am currently building a flask app that will have lots of data that I think I cannot load into memory. I have searched many places, and have found the SQL seems to be a good solution. Sadly I cannot use SQL for this project due to some limitations of SQL.

My project consists many entries of

database[lookupkey1][lookupkey2]...and more lookup keys

My current plan is to override __getitem__ and __setitem__ and __delitem__ and replace them with calls to the database. Is their any kind of database that can store large amounts of maps/dictionaries like

{"spam":{"bar":["foo","test"],"foo":["bar"]}}

I am also currently using JSON to save data, so it would be appreciated if the database had a easy way to migrate my current database.

Sorry that I'm not very good at writing stack overflow questions.

I'd be quite surprised if SQL has enough limitations to seriously hamper your project. Which implementation of SQL are you using that's not meeting your needs? — BallpointBen
– BallpointBen, Commented Dec 21, 2018 at 6:42
SQLite for Python 3, also I'm pretty bad at SQL anyway too so I'm taking some tutorials. — Raymond
– Raymond, Commented Dec 21, 2018 at 6:44

Fine · Accepted Answer · 2018-12-21 15:30:56Z

1

Most document-oriented DBs like MongoDB would allow you to save data as nested dict-list-like objects and query them using their keys and indexes.

P.S. Accessing such a DB through a Python's dict accessors is a bad idea as it would produce a redundant DB query for each step which is highly ineffective and may lead to performance problems. Try looking at ORM for a DB you choose as most ORMs would allow you to access document-oriented DB's data in a way similar to accessing dicts and lists.

answered Dec 21, 2018 at 15:30

Fine

2,1641 gold badge14 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Raymond Over a year ago

I'm going to try your solution, and thanks for the tip about dict accessors!

Collectives™ on Stack Overflow

Python 3 : Storing Data without loading it into memory

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related