How to replace large python dictionary with a memory efficient data structure?

Question

I use a python dictionary to store key value pairs and the dictionary gets too large(>100GB) and hits a memory limit.

What is a better memory efficient data structure to store key value pairs in python? E.g. we can use generators to replace lists

Taron Qalashyan · Accepted Answer · 2021-06-12 14:11:58Z

2

You can use sqlitedict which provides key-value interface to SQLite database. About memory usage. SQLite doesn't need your dataset to fit in RAM. By default it caches up to cache_size pages, which is barely 2MB .

answered Jun 12, 2021 at 14:11

Taron Qalashyan

7214 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

dagnelies · Accepted Answer · 2021-12-13 12:11:50Z

0

Maybe that could help: https://github.com/dagnelies/pysos

It keeps only the index in memory and keeps the data on disk.

answered Dec 13, 2021 at 12:11

dagnelies

5,3457 gold badges40 silver badges58 bronze badges

Collectives™ on Stack Overflow

How to replace large python dictionary with a memory efficient data structure?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related