ValueError when apply function to data frame in python-pandas

Question

Let's say I have a very simple data frame:

import pandas as pd
df = pd.DataFrame(np.full((6), 1))

Now I am going to define a function that generate a numpy array with random length and add the given value to the tail:

import numpy as np
def func(row):
    l = np.full((np.random.random_integer(5)), 1)
    return np.hstack(l, row)

When I try to apply the function to df to get a 2-D array:

df.apply(func, axis=1),

I got such an error:

ValueError: Shape of passed values is (6, 2), indices imply (6, 1)

Do you know what is the problem and how to fix it? Thank you in advance!

random_integet is not a np function can you update your code with something that runs — EdChum
– EdChum, Commented Jun 29, 2016 at 7:51

EdChum · Accepted Answer · 2016-06-29 08:19:10Z

1

Firstly you want np.random.random_integers, secondly hstack takes a tuple so pass a tuple, thirdly you need to return something that it can align with so in this case a Series:

In [213]:
df = pd.DataFrame(np.full((6), 1))
def func(row):
    l = np.full((np.random.random_integers(5)), 1)
    return pd.Series(np.hstack((l, row)))

In [214]:    
df.apply(func, axis=1)

Out[214]:
     0    1    2    3    4    5
0  1.0  1.0  1.0  NaN  NaN  NaN
1  1.0  1.0  NaN  NaN  NaN  NaN
2  1.0  1.0  NaN  NaN  NaN  NaN
3  1.0  1.0  1.0  NaN  NaN  NaN
4  1.0  1.0  1.0  1.0  1.0  NaN
5  1.0  1.0  1.0  1.0  1.0  1.0

Note that I get a ton of warnings about the above:

C:\WinPython-64bit-3.4.3.5\python-3.4.3.amd64\lib\site-packages\numpy\core\numeric.py:301: FutureWarning: in the future, full(3, 1) will return an array of dtype('int32')
  format(shape, fill_value, array(fill_value).dtype), FutureWarning)
C:\WinPython-64bit-3.4.3.5\python-3.4.3.amd64\lib\site-packages\numpy\core\numeric.py:301: FutureWarning: in the future, full(2, 1) will return an array of dtype('int32')
  format(shape, fill_value, array(fill_value).dtype), FutureWarning)
C:\WinPython-64bit-3.4.3.5\python-3.4.3.amd64\lib\site-packages\numpy\core\numeric.py:301: FutureWarning: in the future, full(1, 1) will return an array of dtype('int32')
  format(shape, fill_value, array(fill_value).dtype), FutureWarning)
C:\WinPython-64bit-3.4.3.5\python-3.4.3.amd64\lib\site-packages\numpy\core\numeric.py:301: FutureWarning: in the future, full(4, 1) will return an array of dtype('int32')
  format(shape, fill_value, array(fill_value).dtype), FutureWarning)
C:\WinPython-64bit-3.4.3.5\python-3.4.3.amd64\lib\site-packages\numpy\core\numeric.py:301: FutureWarning: in the future, full(5, 1) will return an array of dtype('int32')
  format(shape, fill_value, array(fill_value).dtype), FutureWarning)

To get a np array from a df call attribute values:

df.apply(func, axis=1).values

edited Jun 29, 2016 at 8:19

answered Jun 29, 2016 at 7:58

EdChum

397k204 gold badges836 silver badges583 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

user5779223 Over a year ago

Thanks for your magic answer! But can I transfer it to numpy 2-D array or matrix? Thanks!

EdChum Over a year ago

Can you explain what you mean by editing your question with desired output

user5779223 Over a year ago

I have edited the question already. Actually I just want the output is a numpy array instead of series. Thanks.

EdChum Over a year ago

So you want df.apply(func, axis=1).values?

user5779223 Over a year ago

Yes! Thanks a lot!

Collectives™ on Stack Overflow

ValueError when apply function to data frame in python-pandas

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related