1

I am retrieving structured numerical data (float 2-3 decimal spaces) via http requests from a server. The data comes in as sets of numbers which are then converted into an array/list. I want to then store each set of data locally on my computer so that I can further operate on it.

Since there are very many of these data sets which need to be collected, simply writing each data set that comes in to a .txt file does not seem very efficient. On the other hand I am aware that there are various solutions such as mongodb, python to sql interfaces...ect but i'm unsure of which one I should use and which would be the most appropriate and efficient for this scenario.

Also the database that is created must be able to interface and be queried from different languages such as MATLAB.

Improve this question
2
  • What kind of data are you dealing with? How large is it? Commented May 16, 2016 at 3:48
  • temperature data. Each data set represents one full 24 hour period of measurement. overall there are roughly about 3 million datasets each with about 500-1000 numbers in each set depending on the frequency of measurement for that particular set @AnthonyE Commented May 16, 2016 at 4:07

3 Answers 3

Reset to default
1

If you just want to store it somewhere so MATLAB can work with it; pick your choice from the databases supported by matlab and then install the appropriate drivers for Python for that database.

All databases in Python have a standard API (called the dbapi) so there is a uniform way of dealing with databases.

As you haven't told us how to intend to work with this data later on, it is difficult to provide any further specifics.

the idea is that i wish to essentially download all of the data onto my machine so that i can operate on it locally later (run analytics and perform certain mathematical operations on it) instead of having to call it from the server constantly.

For that purpose you can use any storage mechanism from text files to any of the databases supported by MATLAB - as all databases supported by MATLAB are supported by Python.

You can choose to store the data as "text" and then do the numeric calculations on the application side (ie, MATLAB side). Or you can choose to store the data as numbers/float/decimal (depending on the precision you need) and this will allow you to do some calculations on the database side.

If you just want to store it as text and do calculations on the application side then the easiest option is mongodb as it is schema-less. You would be storing the data as JSON - which may be the format that it is being retrieved from the web.

If you wish to take advantages of some math functions or other capabilities (for example, geospatial calculations) then a better choice is a traditional database that you are familiar with. You'll have to create a schema and define the datatypes for each of your incoming data objects; and then store them appropriately in order to take advantage of the database's query features.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

the idea is that i wish to essentially download all of the data onto my machine so that i can operate on it locally later (run analytics and perform certain mathematical operations on it) instead of having to call it from the server constantly. @BurhanKhalid
thanks for the response, when you say traditional database what do you mean? can you give an example? @BurhanKhalid
Oracle, SQL, Postgresql, etc. The relational databases.
1

I can recommend using a lightweight ORM like peewee which can use a number of SQL databases as the storage method. Then it becomes a matter of choosing the database you want. The simplest database to use is sqlite, but should you decide that's not fast enough switching to another database like PostgreSQL or MySQL is trivial.

The advantage of the ORM is that you can use Python syntax to interact with the SQL database and don't have to learn any SQL.

Improve this answer

Comments

1

Have you considered HDF5? It's very efficient for numerical data, and is supported by both Python and Matlab.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.