TypeError when using `df.apply` (Pandas)

Question

I have a pandas dataframe that looks like this:

I want to take the log of each value in the dataframe.

So that seemed like no problem at first, and then: data.apply(lambda x:math.log(x)) returned a type error (cannot convert series to class 'float').

Okay, fine--so, while type checking is often frowned upon, I gave it a shot (also tried casting x to a float, same problem):

isinstance((data['A1BG'][0]), np.float64) returns true, so I tried:

data.apply(lambda x: math.log(x) if isinstance(x, np.float64) else x). That ran without any errors, but it didn't change any values in my dataframe.

What am I doing wrong?

Thanks!

Is the Hybridization column an index? If not, set it as the index first and then call applymap. — cs95
– cs95, Commented Aug 2, 2017 at 13:51
Yes--earlier in the notebook: data = data.set_index("Hybridization REF") — julianstanley
– julianstanley, Commented Aug 2, 2017 at 13:52

cs95 · Accepted Answer · 2017-08-02 13:58:06Z

2

What happens is that df.apply returns a pd.Series object for the lambda to operate over... It basically operates over a Series at a time, not one float at a time.

So, with

data.apply(lambda x: math.log(x) if isinstance(x, np.float64) else x)

isinstance(x, np.float64) is never true (because x is a pd.Series type) and so the else is always executed.

To remedy this, you can operate a column at a time, using df.applymap:

data.applymap(math.log)

Using apply, the solution is similar, but you cannot escape the lambda:

data.apply(lambda x: np.log(x))

Or, alternatively (pd 0.20):

data.transform(lambda x: np.log(x))

Coincidentally, df.applymap is the fastest, followed by df.apply and df.transform.

edited Aug 2, 2017 at 13:58

answered Aug 2, 2017 at 13:54

cs95

406k106 gold badges744 silver badges797 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

julianstanley Over a year ago

That's a great solution, thank you! Ysearka's solution also works, but this one gives me something that will be more applicable to a wider range of future uses.

cs95 Over a year ago

@JulianStanley You can actually shorten your expression. Just pass the callback without a lambda - applymap(callback). Pandas will vectorise it automatically.

cs95 Over a year ago

@JulianStanley Just tested on bigger data, and you should stick to apply or transform for large data. Cheers.

ysearka · Accepted Answer · 2017-08-02 13:59:09Z

1

When you do apply on a dataframe, the apply function will be cast upon a Pandas.Series not a float (opposing to when you use apply on a Series). Then instead of math.log you should use np.log)

EDIT:

With examples it's always better:

test = pd.DataFrame(columns = ['a','b'])
test.a = np.random.random(5)
test.b = np.random.random(5)

    a           b
0   0.430111    0.420516
1   0.367704    0.785093
2   0.034130    0.839822
3   0.310254    0.755089
4   0.098302    0.136995

If you try the following, it won't work:

test.apply(lambda x: math.log(x))

TypeError: ("cannot convert the series to <class 'float'>", 'occurred at index a')

But this will do the job:

test.apply(lambda x: np.log(x))

    a           b
0   -0.843711   -0.866273
1   -1.000476   -0.241953
2   -3.377588   -0.174565
3   -1.170364   -0.280919
4   -2.319708   -1.987811

edited Aug 2, 2017 at 13:59

answered Aug 2, 2017 at 13:53

ysearka

3,8655 gold badges24 silver badges42 bronze badges

1 Comment

julianstanley Over a year ago

Bam. Quick, simple, and worked like a charm. Thanks much!

kanav anand · Accepted Answer · 2017-08-02 13:54:12Z

0

Try this

 import math
 data.apply(lambda x:math.log(list(x)))

answered Aug 2, 2017 at 13:54

kanav anand

4202 gold badges4 silver badges15 bronze badges

1 Comment

julianstanley Over a year ago

This also works, for a similar reason as ColdSpeed's solution--changing an array to a list. Thanks for the help!

Collectives™ on Stack Overflow

TypeError when using `df.apply` (Pandas)

3 Answers 3

3 Comments

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related