Create numpy array from generator, list of lists

Question

I am trying to convert a generator to a numpy array. I apply a map function on a list of data and the result is a generator. I tried doing list(map()) and then creating the numpy vector but it takes a long time. I saw somewhere that I can directly use np.fromiter to create a numpy vector from my generator. However, I run into this error:

ValueError: setting an array element with a sequence.

I've found out that the error rises because my generator generates a list of lists. like: [[1,2,3], [4,5,6]] and I should use a proper structural dtype for the fromiter() function. I couldn't find a proper explanation of the way to do this. Can you help me?

Here's a full example:

import numpy as np

def foo(bar):
  return [bar] * 3 # so for 4 it returns [4,4,4], ..

a = [1,2,3,4,5,6,7]
b = map(foo,a)
c = np.fromiter(b, int) # this doesn't work.

Please post an MCVE

Mad Physicist
– Mad Physicist

2020-04-28 17:16:37 +00:00
Commented Apr 28, 2020 at 17:16 — Mad Physicist
– Mad Physicist, Commented Apr 28, 2020 at 17:16
ok. I'll edit the post ASAP.

P.Alipoor
– P.Alipoor

2020-04-28 17:20:55 +00:00
Commented Apr 28, 2020 at 17:20 — P.Alipoor
– P.Alipoor, Commented Apr 28, 2020 at 17:20

hpaulj · Accepted Answer · 2020-04-28 17:35:27Z

2

To use a compound dtype, the function has to return tuples, not lists

In [977]: def foo(bar): 
     ...:   return (bar,) * 3 # so for 4 it returns [4,4,4], .. 
     ...:  
     ...: a = [1,2,3,4,5,6,7] 
     ...: b = map(foo,a)                                                                               
In [978]: list(b)                                                                                      
Out[978]: [(1, 1, 1), (2, 2, 2), (3, 3, 3), (4, 4, 4), (5, 5, 5), (6, 6, 6), (7, 7, 7)]
In [979]: def foo(bar): 
     ...:   return (bar,) * 3 # so for 4 it returns [4,4,4], .. 
     ...:  
     ...: a = [1,2,3,4,5,6,7] 
     ...: b = map(foo,a)                                                                               
In [980]: np.fromiter(b, 'i,i,i')                                                                      
Out[980]: 
array([(1, 1, 1), (2, 2, 2), (3, 3, 3), (4, 4, 4), (5, 5, 5), (6, 6, 6),
       (7, 7, 7)], dtype=[('f0', '<i4'), ('f1', '<i4'), ('f2', '<i4')])

some timings:

In [981]: %%timeit b = map(foo,a) 
     ...: np.array(list(b)) 
     ...:  
     ...:                                                                                              
1.9 µs ± 55.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In [982]: %%timeit b = map(foo,a) 
     ...: np.fromiter(b, 'i,i,i') 
     ...:  
     ...:                                                                                              
17.2 µs ± 9.72 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

answered Apr 28, 2020 at 17:35

hpaulj

233k14 gold badges260 silver badges392 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

P.Alipoor Over a year ago

Thanks! So it isn't faster as I thought.

hpaulj Over a year ago

Generators are most useful when you string several together. They delay evaluation, but soon or later you have to generate all values.

Collectives™ on Stack Overflow

Create numpy array from generator, list of lists

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related