Python dtype('O') . Processing object data type. Converting to string/integer

Question

I have the table from ImDB with actors.

From this table I want to drop all rows where imdb_actors.birthYear is missing or is less than 1950 and also drope those where imdb_actors.deathYear has some value.

Idea is to get a dataset with actors who are alive and not retired.

imdb_actors.birthYear.dtype
Out:dtype('O')

And I can't convert to string, this doesn't help: imdb_actors['birthYear'] = imdb_actors['birthYear'].astype('|S'). It just ruins all years.

That's why I can't execute: imdb_actors[imdb_actors.birthYear >= 1955] When I try imdb_actors.birthYear.astype(str).astype(int) I get the message: ValueError: invalid literal for int() with base 10: '\\N'

What will be the way to drop missing and apply >= 1950 condition?

jpp · Accepted Answer · 2018-09-05 15:37:22Z

2

First convert numeric data to numeric series:

num_cols = ['birthYear', 'deathYear']
df[num_cols] = df[num_cols].apply(pd.to_numeric, errors='coerce')

Specifying errors='coerce' forces non-convertible elements to NaN.

Then create masks for your 3 conditions, combine via the vectorised | "or" operator, negate via ~, and apply Boolean indexing on your dataframe:

m1 = df['birthYear'].isnull()
m2 = df['birthYear'] < 1950
m3 = df['deathYear'].notnull()

res = df[~(m1 | m2 | m3)]

answered Sep 5, 2018 at 15:37

jpp

166k37 gold badges301 silver badges363 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Pinky the mouse Over a year ago

df[num_cols] = df[num_cols].apply(pd.to_numeric, errors='coerce') loses good cells with years and just gives me back each column as NaN.

jpp Over a year ago

I'd love to check this out, but I'm having trouble copy-pasting the image you posted on your question into my code. See also How to make good reproducible pandas examples.

Alex · Accepted Answer · 2018-09-05 15:32:17Z

0

Your problem is that the type of your birthYear serie is Object which would be for strings or a mix of types.

You will want to clean this serie first by applying a function like this :

imdb_actors.birthYear = imdb_actors.birthYear.map(lambda x: int(x) if str(x) != '\\N' else pd.np.nan)

then you can do your filtering:

imdb_actors[imdb_actors.birthYear >= 1955]

answered Sep 5, 2018 at 15:32

Alex

8265 silver badges14 bronze badges

5 Comments

Pinky the mouse Over a year ago

Trying to run the first line I get the error: ValueError: invalid literal for int() with base 10: b'\\N'

Alex Over a year ago

could you post a sample of your data ?

jpp Over a year ago

You should use vectorised operations. map + lambda on object dtype is no better than a simple loop with a list.

Pinky the mouse Over a year ago

I have a great trouble converting tables to a nice format on SO. Is there a way to paste it nicely? Sorry for silly question.

jpp Over a year ago

@Pinkythemouse, That's a good question. See this question for answers: How to make good reproducible pandas examples.

Collectives™ on Stack Overflow

Python dtype('O') . Processing object data type. Converting to string/integer

2 Answers 2

2 Comments

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related