Fast way to populate numpy array with multiple iterations of multiple numbers

Question

I need to populate a numpy array, execution speed is important to me. The array will use a dictionary which will specify how many instances (indexed key value) of the array value (key value) I need.

The script below is my attempt, testing shows it takes 0.14 seconds to run but if I remove the hstack it runs in 0.004 s. So I conclude that it is the concatenating of the array that is taking the time. What's a better method?

Note the dictionary below is just a test case, in general I will have about a 100 different values and each value will repeat approximately 10,000 times.

td = {}
for ii in range(100):
    td[ii] = 10000+ii
a = np.ones(0)
for aa in td:
    a = np.hstack((a,np.ones(td[aa])*aa))

Paul · Accepted Answer · 2014-08-29 03:24:48Z

3

It's almost another 10x faster (than Josh's solution) to just flat out preallocate your memory.

a = np.empty((sum(td.values(),)))
i=0
for k,v in td.iteritems():
    a[i:i+v]=k
    i +=v

Why mess around with intermediate storage when you have enough info at the start to size your array? (np.empty is a quick way to size an array without actually setting any values yet)

answered Aug 29, 2014 at 3:24

Paul

43.9k17 gold badges112 silver badges126 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

JoshAdel · Accepted Answer · 2014-08-29 02:42:19Z

1

This code does the same thing but takes 3 ms vs 200 ms on my machine:

td = {}
for ii in range(100):
    td[ii] = 10000+ii

a = np.hstack([np.ones(td[aa])*aa for aa in td])

It calls np.hstack once on a list of arrays rather than repeatedly joining. Also, note that the order that you iterate through a dictionary is not guaranteed to be in the same order of insertion (use an OrderedDict if you want that), so you should be careful.

answered Aug 29, 2014 at 2:42

JoshAdel

69.1k27 gold badges146 silver badges146 bronze badges

Collectives™ on Stack Overflow

Fast way to populate numpy array with multiple iterations of multiple numbers

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related