0

Assume that I have a big float numpy array:

How to save this float numpy array to a binary file with less storage using numpy.save?

np.save(nucleosomenpyOutputFilePath,averageSignalArray)
5
  • 1
    What's the current dtype? Commented Feb 3, 2019 at 20:50
  • I got this numpy array by the following statement averageSignalArray=np.divide(accumulatedSignalArray,accumulatedCountArray) I guess its dtype=float Commented Feb 3, 2019 at 20:52
  • 1
    Do you know what I mean by dtype? Commented Feb 3, 2019 at 20:54
  • Yes, of course dtype. Commented Feb 3, 2019 at 21:08
  • There's float32 and float64, 4 and 8 bytes per element. Commented Feb 3, 2019 at 21:48

3 Answers 3

1

Thanks @hpaulj. You openedmy eyes.

By playing with dtype=np.float32 or dtype=np.float16 in the following statements

averageSignalArray=np.divide(accumulatedSignalArray,accumulatedCountArray,dtype=np.float32)
averageSignalArray=np.divide(accumulatedSignalArray,accumulatedCountArray,dtype=np.float16)

I got different nparrays and save them in the following step:

np.save(nucleosomenpyOutputFilePath,averageSignalArray)
Sign up to request clarification or add additional context in comments.

Comments

1

If your goal is to just save size on the resulting files and you can install additional python packages and use compressed arrays. https://github.com/Blosc/bcolz

Comments

1

Probably one of the fastest and most space-efficient ways of doing this is by using Bloscpack:

https://github.com/blosc/bloscpack

You can read about using the Python API here:

https://github.com/blosc/bloscpack#python-api

And lastly, here is an example:

>>> import numpy as np
>>> import bloscpack as bp
>>> a = np.linspace(0, 1, 3e8)
>>> print a.size, a.dtype
300000000 float64
>>> bp.pack_ndarray_to_file(a, 'a.blp')
>>> b = bp.unpack_ndarray_from_file('a.blp')
>>> (a == b).all()
True

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.