Storing an integer less than 256 within a byte of memory in Python

Question

I have a huge array (of arrays) of integers in the range 0-255. Since I know the range of the integers, hence I want to optimize the space occupied by them by storing each integer within a single byte.

In C++, I would simply use char to store the integers, but I am not able to find a way out in Python.

>>> a = 10
>>> sys.getsizeof(a)
24
>>> b = chr(a)
>>> sys.getsizeof(b)
38
>>> c = bytearray(1)
>>> c[0] = b
>>> c[0]
10
>>> sys.getsizeof(c[0])
24
>>> c
bytearray(b'\n')
>>> sys.getsizeof(c)
50

I have searched for data types available in Python, but I am not able to get any data type which can give me sys.getsizeof() equal to 1. I want to know whether there exists a spatially optimal way of storing such integers.

There are compact arrays in the standard array module. But if you also want fast operations on your data, you might as well use Numpy. — PM 2Ring
– PM 2Ring, Commented Mar 2, 2016 at 19:37

user2357112 · Accepted Answer · 2016-03-02 18:50:11Z

4

sys.getsizeof(c[0]) doesn't report the actual amount of memory used to store the first element of c. Accessing c[0] makes Python construct an integer object (or fetch one from the small integer cache) to represent the value, but the bytearray does store the value as one byte.

This is more obvious with a larger bytearray:

>>> sys.getsizeof(bytearray([5]*1000))
1168

You can see that this bytearray couldn't possibly be using more than 1 byte per element, or it would be at least 2000 bytes in size. (The excess space is due to overallocation to accommodate additional elements, and some object overhead.)

answered Mar 2, 2016 at 18:50

user2357112

286k32 gold badges490 silver badges570 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Maxim Egorushkin · Accepted Answer · 2016-03-02 18:58:10Z

2

You can use numpy arrays for that. E.g.:

import numpy as np

byte_array = np.empty(10, np.uint8) # an array of 10 uninitialized bytes

See other numpy array constructors for more details.

edited Mar 2, 2016 at 18:58

answered Mar 2, 2016 at 18:54

Maxim Egorushkin

138k19 gold badges201 silver badges293 bronze badges

2 Comments

Alex Over a year ago

That's not an array of 10 bytes. It has a single (1 byte) element.

Maxim Egorushkin Over a year ago

@Alex Mistyped the constructor function.

Steve Barnes · Accepted Answer · 2016-03-02 18:54:57Z

1

If you are dealing with huge arrays then you will probably be best off using numpy which includes a lot of array tools for you.

There is some overhead but it is minimal:

import numpy as np
import sys

a = np.array([0]*10000, np.uint8)    
len(a)
# 10000
sys.getsizeof(a)
# 10048
sys.getsizeof(a[0])
# 13
a = np.array([0]*1000000, np.uint8)
sys.getsizeof(a)
# 1000048

answered Mar 2, 2016 at 18:54

Steve Barnes

28.5k6 gold badges68 silver badges80 bronze badges

Comments

recursive · Accepted Answer · 2016-03-02 18:42:02Z

1

There is a bytes class for the purpose of storing a packed sequence of bytes. I don't think there's an easy way of storing just a single number using one byte of memory.

Documentation for bytes

>>> bytes.fromhex('2Ef0 F1f2  ')
b'.\xf0\xf1\xf2'

>>> sys.getsizeof(bytes.fromhex(''))
33
>>> sys.getsizeof(bytes.fromhex('dead'))
35
>>> sys.getsizeof(bytes.fromhex('deadbeef'))
37

answered Mar 2, 2016 at 18:42

recursive

86.5k36 gold badges155 silver badges246 bronze badges

Collectives™ on Stack Overflow

Storing an integer less than 256 within a byte of memory in Python

4 Answers 4

Comments

2 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related