10

Looking for advice on the best technique for saving complex Python data structures across program sessions.

Here's a list of techniques I've come up with so far:

  • pickle/cpickle
  • json
  • jsonpickle
  • xml
  • database (like SQLite)

Pickle is the easiest and fastest technique, but my understanding is that there is no guarantee that pickle output will work across various versions of Python 2.x/3.x or across 32 and 64 bit implementations of Python.

Json only works for simple data structures. Jsonpickle seems to correct this AND seems to be written to work across different versions of Python.

Serializing to XML or to a database is possible, but represents extra effort since we would have to do the serialization ourselves manually.

Thank you, Malcolm

Improve this question

4 Answers 4

Reset to default
15

You have a misconception about pickles: they are guaranteed to work across Python versions. You simply have to choose a protocol version that is supported by all the Python versions you care about.

The technique you left out is marshal, which is not guaranteed to work across Python versions (and btw, is how .pyc files are written).

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Ned: Thank you for pointing out my confusion between pickling and marshalling.
4

You left out the marshal and shelve modules.

Also this python docs page covers persistence

Improve this answer

1 Comment

SpliFF: Thanks for the link to the Python Persistence web page.
2

Have you looked at PySyck or pyYAML?

Improve this answer

1 Comment

Micholson: I had forgotten about pyYAML. Looks like an interesting compromise between JSON (doesn't work with complex data structures) and pickle. Have you looked at the jsonpickle project. Very impressive as well.
2

What are your criteria for "best" ?

  • pickle can do most Python structures, deeply nested ones too
  • sqlite dbs can be easily queried (if you know sql :)
  • speed / memory ? trust no benchmarks that you haven't faked yourself.

(Fine print:
cPickle.dump(protocol=-1) compresses, in one case 15M pickle / 60M sqlite, but can break.
Strings that occur many times, e.g. country names, may take more memory than you expect; see the builtin intern().
)

Improve this answer

1 Comment

Denis: Thanks for your warning about protocol=-1 and sense of humor (re: trust no benchmarks you haven't faked yourself ... LMAO!)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.