1

I'm an intermediate python developer with quite a bit of experience working with t-sql/mysql in data analytics. I'm trying my hand at some web programming using python to build certain components of the back-end of my site.

In one of my applications, instead of querying a database, I've placed some dictionary-based data structures directly in a list in a python file, and I import it into the script that holds the application logic. Important to note that this data is static and around 5k dictionaries at this point.

The data itself contains key-value pairs where the values is frequently a list of a tuple, so I find the flexibility of the python data structure to be easier to work with than a traditional RDBMS table.

I've scoured the internet and I can't seem to find any reference to developers actually using a .py file to store data for use in their programs, which leads me to believe this method shouldn't be used in a production site.

I'm hoping for some clarity around why this might be the case (and potential alternatives)

  1. Will my .py solution scale poorly vs a RDBMS solution? Either as the number of users using my application grows or if I choose to support more vehicles in my data set?
  2. Is there a nosql solution that might give me the flexible data structure I need if the .py solution doesn't work? Or is a mysql solution still superior in this use case? Thanks!
Improve this question
1
  • Maybe you can use a JSON file format. Or are you aware of the sqlalchemy package which do object mapping of an RDBMS (e. g. sqlite3). Commented Jan 12, 2018 at 4:12

2 Answers 2

Reset to default
1

I've done something similar in the past - shelving. This is a standard lib module that allows you to write python objects to disk, making them nice and easy to load.

But as with your solution, there would be significant scalability issues. For example - in the shelving example the entire object gets loaded into memory. I suspect the same of your implementation, assuming you import it to get to the data.

Among the advantages of a database is that there is no requirement to keep the entire database in memory. It also means that you can perform certain searches much more quickly; iterating through thousands of objects in python to check them for a certain condition will probably be quite slow compared to a well written database query.

If your current solution still works then there is probably no real reason to change everything right away. But if you think this application is going to continue to grow, then having a migration plan is not a bad idea.

I ended up moving my application to RethinkDB for what it's worth.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

I can say that your .py file solution is probably not going to last you for much longer. If you're looking for a suitable key-value store solution you should definitely check out nosql database backends such as redis or mongodb.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.