using file/db as the buffer for very big numpy array to yield data prevent overflow?

Question

In using the numpy.darray, I met a memory overflow problem due to the size of data，for example:

Suppose I have a 100000000 * 100000000 * 100000000 float64 array data source, when I want to read data and process it in memory with np. It will raise a Memoray Error because it works out all memory for storing such a big array in memory.

Then maybe using a disk file / database as a buffer to store the array is a solution, when I want to use data, it will get the necessary data from the file / database, otherwise, it is just a python object take few memory.

Is it possible write such a adapter?

Thanks.

Rgs, KC

Peter Prettenhofer · Accepted Answer · 2010-09-29 13:21:26Z

1

Take a look at pytables or numpy.memmap, maybe they fit your needs.

best, Peter

answered Sep 29, 2010 at 13:21

Peter Prettenhofer

1,97120 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

K. C Over a year ago

thanks, however, they are not, I need to write a sqlite based numpy.array reader for the SQL language, the most suitable is ATpy, however, it can not "lazy read"...

baklarz2048 · Accepted Answer · 2010-09-29 05:49:52Z

0

If You have matrices with lots of zeros use scipy.sparse.csc_matrix. It's possible to write everything, for example You can override numarray array class.

answered Sep 29, 2010 at 5:49

baklarz2048

11k2 gold badges33 silver badges38 bronze badges

1 Comment

K. C Over a year ago

thanks, that may not what I need, what I want to do is something like linux command "cat big_file | more" then I can usin page function to seek from different pages.

Collectives™ on Stack Overflow

using file/db as the buffer for very big numpy array to yield data prevent overflow?

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related