7

I am working on a personal project in Python where I need some form of persistent data. The data would fit in 2-3 tables of 10-20 columns and 100-200 records each. I have a basic understanding of SQL, so a database seems to make some sense.

I am new to Python, so I am not familiar with the options for database interface from Python. I have also heard about pickling and am not sure if that would be a better solution for my project size. Can anyone recommend a good solution?

Improve this question

6 Answers 6

Reset to default
9

Or, if you just want to persist data between executions - for such a small data set you could have a look at the pickle module for persistency, and just load the data into memory during execution.

It's a simple solution - but for a personal project it might be enough.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

+1 I'd also start with pickle, just throw a bunch of lists of tuples at it.
I think I will start with pickle. Thanks to all for the suggestions and information about different solutions!
6

You should use sqlite3 module for this, it is included in Python.

Also you may want too look for an ORM solution.

Improve this answer

Comments

3

This sounds like very few data. An SQL DB might be overkill, especially with an ORM on top. I'd check whether JSON could do the job...

Improve this answer

5 Comments

what are the advantages of JSON over pickle? From my cursory understanding, it seems like it would be better to use Python's own pickle format.
It's widely supported (i.e. practically platform/language-independent), plus it's human readable.
But Json is not persistent on its own. You need to write and read it to a file/files, serialize/deserialize on reads/writes. Pickling might be faster on that. But pickling really pays when you have more complex data types to persist, like class instances.
@ThomasH Agreed. In case of small, simple data I would however opt for json - I just like the flexibility that comes with txt files, plus I might be a bit biased anyway as I use JS at times in addition...
json is pretty much the exact same code as pickle, so I wouldn't necessarily choose one because it's "easier" or anything. Pickle can handle more types, but it's not as human readable as json and it's limited to python. You could also consider YAML, which is similar to json but even more human readable (IMO).
1

I agree with using sqlite3. It is very easy to use, you don't need to worry about having to set up a database server. You should check out the SQLAlchemy library too.

Improve this answer

Comments

1

The real question is really what kind of operations you want to do with your data.

As far as storage possibilities, the simplest solutions are indeed sqlite3 and pickle.

The solution that you will choose depends basically on whether using SQL or Python is the easiest way for you to manage your data. SQL is probably better at complex operations than Python, but Python is definitely more lightweight and simpler, and therefore is a good choice for simple operations. So, if using pickle+Python is too cumbersome, then sqlite3 is a very good choice.

Improve this answer

Comments

0

Peewee is another ORM that works with SQLite. It is an alternative to SQLAlchemy. If using SQLite, I would consider Peewee for pet projects and SQLAlchemy for professional work. I typically would not use SQLite directly.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.