Concatenate nested list of array with partial empty sublist

Question

The objective is to concatenate nested list of arrays (i.e., list_arr). However, some of the sublists within the list_arr is of len zero.

Simply using np.array or np.asarray on the list_arr does not produce the intended result.

import numpy as np

ncondition=2
nnodes=30
nsbj=6


np.random.seed(0)

# Example of nested list list_arr

list_arr=[[[np.concatenate([[idx_sbj],[ncondi],[nepoch] ,np.random.rand(nnodes)]) for nepoch in range(np.random.randint(5))] \
 for ncondi in range(ncondition)] for idx_sbj in range(nsbj)]

The following does not produce the expected concatenate output

test1=np.asarray(list_arr)
test2=np.array(list_arr)
test3= np.vstack(list_arr)

The expected output is an array of shapes (15,33)

what's the problem with test2 and test3. test1 is the same as test2. — hpaulj
– hpaulj, Commented Nov 21, 2021 at 15:48
Thanks for dropping by @hpaulj. I agree It is the same, just to show what has been tested — rpb
– rpb, Commented Nov 21, 2021 at 15:53
OK I randyour code (I shouldn't have needed to), and found that it makes nested lists of various sizes. np.array produces an object dtype array with the ragged warning. vstack fails, saying that the dimensions don't match. You should have told us that!. The outer len is 6. There are arrays a couple layers down, but also some [] lists. This cannot be made into a regular numpy array. I'm voting to close — hpaulj
– hpaulj, Commented Nov 21, 2021 at 17:01

hpaulj · Accepted Answer · 2021-11-21 18:03:46Z

OK, my curiosity got the better of me.

Make an object dtype array from the list:

In [280]: arr=np.array(list_arr,object)
In [281]: arr.shape
Out[281]: (6, 2)

All elements of this array are lists, with len:

In [282]: np.frompyfunc(len,1,1)(arr)
Out[282]: 
array([[4, 1],
       [0, 2],
       [0, 2],
       [0, 0],
       [2, 3],
       [1, 0]], dtype=object)

Looking at specific sublists. One has two empty lists

In [283]: list_arr[3]
Out[283]: [[], []]

others have one empty list, either first or second:

In [284]: list_arr[-1]
Out[284]: 
[[array([5.        , 0.        , 0.        , 0.3681024 , 0.3127533 ,
         0.80183615, 0.07044719, 0.68357296, 0.38072924, 0.63393096,
         ...])],
 []]

and some have lists of differing numbers of arrays:

If I add up the numbers in [282] I get 15, so that must be where you get the (15,33). And presumably all the arrays have the same length.

The outer layer of nesting isn't relevant, so we can ravel and remove it.

In [295]: alist = arr.ravel().tolist()

then filter out the empty lists, and apply vstack to the remaining:

In [296]: alist = [np.vstack(x) for x in alist if x]
In [297]: len(alist)
Out[297]: 7

and one more vstack to join those:

In [298]: arr2 = np.vstack(alist)
In [299]: arr2.shape
Out[299]: (15, 33)

Collectives™ on Stack Overflow

Concatenate nested list of array with partial empty sublist

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related