Initialise numpy array of unknown length

Question

I want to be able to 'build' a numpy array on the fly, I do not know the size of this array in advance.

For example I want to do something like this:

a= np.array()
for x in y:
     a.append(x)

Which would result in a containing all the elements of x, obviously this is a trivial answer. I am just curious whether this is possible?

What may be a more efficient approach is to allocate some large array, and double the size of it every time you reach capacity. — wim
– wim, Commented Apr 24, 2013 at 1:09
What is the rule that tells you the values to put into the array? What is the rule that tells you what the size of the final array will be, as you are getting the data? This question doesn't appear to describe an adequately clear problem statement. — Karl Knechtel
– Karl Knechtel, Commented Sep 19, 2023 at 22:09

Fred Foo · Accepted Answer · 2012-04-12 11:04:00Z

129

Build a Python list and convert that to a Numpy array. That takes amortized O(1) time per append + O(n) for the conversion to array, for a total of O(n).

    a = []
    for x in y:
        a.append(x)
    a = np.array(a)

answered Apr 12, 2012 at 11:04

Fred Foo

365k80 gold badges765 silver badges852 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

A T Over a year ago

Or better still: a = np.array([x for x in y]); or just a = np.array(list(y))

alexisdm · Accepted Answer · 2012-04-12 10:57:33Z

19

You can do this:

a = np.array([])
for x in y:
    a = np.append(a, x)

answered Apr 12, 2012 at 10:57

alexisdm

29.9k6 gold badges68 silver badges102 bronze badges

4 Comments

Fred Foo Over a year ago

That takes linear time per append.

Alex Gaudio Over a year ago

This approach copies the array every append, which is O(sum(range(n))). On my laptop, this method was 42 times slower than @larsman's method: Building a list following larsmans method exactly takes me 1000 loops, best of 3: 1.53 ms per loop . Following this method exactly takes me 10 loops, best of 3: 64.8 ms per loop.

Marcos Pereira Over a year ago

Code should be readable, not fast, unless speed is your bottleneck. Why optimize code you'll run once?

Prasad Raghavendra Over a year ago

Nice comment @AlexGaudio . I really didn't know that. Thank you!

Mr_and_Mrs_D · Accepted Answer · 2017-04-05 13:14:11Z

7

Since y is an iterable I really do not see why the calls to append:

a = np.array(list(y))

will do and it's much faster:

import timeit

print timeit.timeit('list(s)', 's=set(x for x in xrange(1000))')
# 23.952975494633154

print timeit.timeit("""li=[]
for x in s: li.append(x)""", 's=set(x for x in xrange(1000))')
# 189.3826994248866

edited Apr 5, 2017 at 13:14

answered Mar 27, 2017 at 22:34

Mr_and_Mrs_D

34.5k45 gold badges193 silver badges373 bronze badges

Comments

BenDundee · Accepted Answer · 2013-12-04 13:50:03Z

1

For posterity, I think this is quicker:

a = np.array([np.array(list()) for _ in y])

You might even be able to pass in a generator (i.e. [] -> ()), in which case the inner list is never fully stored in memory.

Responding to comment below:

>>> import numpy as np
>>> y = range(10)
>>> a = np.array([np.array(list) for _ in y])
>>> a
array([array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object),
       array(<type 'list'>, dtype=object)], dtype=object)

edited Dec 4, 2013 at 13:50

answered Apr 24, 2013 at 1:07

BenDundee

4,5613 gold badges32 silver badges35 bronze badges

4 Comments

WestCoastProjects Over a year ago

I made a change here: list(_) and that worked great

BenDundee Over a year ago

To be clear @javadba, you don't need to do that--I'm sure there's some Pythonistas who would take offense :)

WestCoastProjects Over a year ago

This is not a matter of style. without the list(_) it does not even work at last for the case i have that y is an array itself

Akaisteph7 Over a year ago

The OP says "which would result in a containing all the elements of x", so you def do need to do list(_)

chiefenne · Accepted Answer · 2016-08-08 21:19:54Z

1

a = np.empty(0)
for x in y:
    a = np.append(a, x)

answered Aug 8, 2016 at 21:19

chiefenne

6152 gold badges17 silver badges32 bronze badges

1 Comment

Shubham Agrawal Over a year ago

This won't work when x has a non-scalar dimension. For example, if x = np.ones((3,5)), it will fail.

Shubham Agrawal · Accepted Answer · 2021-03-02 07:56:10Z

0

I wrote a small utility function. (most answers above are good. I feel this looks nicer)

def np_unknown_cat(acc, arr):
  arrE = np.expand_dims(arr, axis=0)
  if acc is None:
    return arrE
  else:
    return np.concatenate((acc, arrE))

You can use the above function as the following:

acc = None  # accumulator
arr1 = np.ones((3,4))
acc = np_unknown_cat(acc, arr1)
arr2 = np.ones((3,4))
acc = np_unknown_cat(acc, arr2)

edited Mar 2, 2021 at 7:56

answered Mar 2, 2021 at 7:47

Shubham Agrawal

1822 silver badges13 bronze badges

Comments

Parmar Aayush · Accepted Answer · 2022-08-10 05:56:22Z

list1 = []
size = 1
option = "Y"
for x in range(size):
    ele = input("Enter Element For List One : ")
    list1.append(ele)
while(option == "Y"):
    option = input("\n***Add More Element Press Y ***: ")
    if(option=="Y"):
        size = size + 1
        for x in range(size):
            ele = input("Enter Element For List Element : ")
            list1.append(ele)
            size = 1
    else:
        break;
print(list1)

Take:

list1 = []    # Store Array Element 
size = 1      # Rune at One Time 
option = "Y"  # Take User Choice

Implementation:

For Loop at a range of size variable which runs one time because size = 1
use While loop
- take input from users if they want to add press "Y" else "N"
- in the while loop use the if else statement if the option is "Y" then increase size with which helps run 2 time array because size = size + 1 run another for loop at a range of size
- else break

Collectives™ on Stack Overflow

Initialise numpy array of unknown length

7 Answers 7

1 Comment

4 Comments

Comments

4 Comments

1 Comment

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

7 Answers 7

1 Comment

4 Comments

Comments

4 Comments

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related