I want to save a numpy 2d array as part of a mongodb document, but the resulting size of the document is 71 MB, well above the 16 MB limit. The numpy array is sparse. Any suggestions on how to compress the numpy 2d array, or save it in a more memory efficient manner? Thanks for your help.
3
-
The idea of stackoverflow is that people do something and ask for clarification. Not crowdsourcing their work for other people.Salvador Dali– Salvador Dali2014-09-27 00:51:06 +00:00Commented Sep 27, 2014 at 0:51
-
I'm doing a project and asking for clarification on a step I'm stuck on. I didn't think it was necessary to display any code to answer this question.kernelmachine– kernelmachine2014-09-27 00:58:13 +00:00Commented Sep 27, 2014 at 0:58
-
1It is hard to add anything else except of mattgemmell.com/what-have-you-tried. You have not done even simple googling. Sparce matrices are not new to a math world and I expected that you will at least google these keywords and try it.Salvador Dali– Salvador Dali2014-09-27 01:04:55 +00:00Commented Sep 27, 2014 at 1:04
Add a comment
|
1 Answer
If your array is sparse, you can easily convert a numpy array to a supported scipy sparse format and back:
http://docs.scipy.org/doc/scipy/reference/sparse.html
You can then save it using the Matrix Market format
http://docs.scipy.org/doc/scipy/reference/generated/scipy.io.mmwrite.html
Or you can save the numpy array directly in a compressed form using savez_compressed:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.savez_compressed.html
I'm not familiar with MongoDB though, so I'm not sure what file formats or objects are supported.
