How to optimize array storage within a numpy array?

Question

I have a numpy array with shape (n, m):

import numpy as np
foo = np.zeros((5,5))

I make some calculations, getting results in a (n, 2) shape:

bar = np.zeros((8,2))

I want to store the calculation results within the array, since I might have to extend them after another calculation. I can do it like this:

foo = np.zeros((5,5), object)

# one calculation result for index (1, 1)
bar1 = np.zeros((8,2))
foo[1, 1] = bar1

# another calculation result for index (1, 1)
bar2 = np.zeros((5,2))
foo[1, 1] = np.concatenate((foo[1, 1], bar2))

however this seems quite odd to me since I have to do a lot of checking if the array has already got a value at this place or not. Additionally I don't know if using object as datatype is a good idea since I only want to store numpy specific data and not any python objects.

Is there a more numpy specific way to this approach?

it looks like using a hashtable (dict in python) of linked lists (list in python) is a better approach. For the hash you could just use 2D->1D array style indexing like [1,1] -> 1*5 + 1 — Hersh Joshi
– Hersh Joshi, Commented Dec 15, 2021 at 15:32

hpaulj · Accepted Answer · 2021-12-15 17:32:54Z

defaultdict streamlines the task of adding values to dict elements incrementallly:

In [644]: from collections import defaultdict

Start with a dict that has default value of list, [].

In [645]: dd = defaultdict(list)
In [646]: dd[(1,1)].append(np.zeros((1,2),int))
In [647]: dd[(1,1)].append(np.ones((3,2),int))
In [648]: dd
Out[648]: 
defaultdict(list,
            {(1, 1): [array([[0, 0]]), array([[1, 1],
                     [1, 1],
                     [1, 1]])]})

Once we've collected all values, we can convert the nested lists into an array:

In [649]: dd[(1,1)] = np.concatenate(dd[(1,1)])
In [650]: dd
Out[650]: 
defaultdict(list,
            {(1, 1): array([[0, 0],
                    [1, 1],
                    [1, 1],
                    [1, 1]])})
In [652]: dict(dd)
Out[652]: 
{(1,
  1): array([[0, 0],
        [1, 1],
        [1, 1],
        [1, 1]])}

In doing the conversion we will have to take care with keys with [], since we can't concatenate an empty list.

Collectives™ on Stack Overflow

How to optimize array storage within a numpy array?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related