Writing to a csv using pandas with filters

Question

I'm using the pandas library to load in a csv file using Python.

import pandas as pd
df = pd.read_csv("movies.csv")

I'm then checking the columns for specific values or statements, such as:

viewNum = df["views"] >= 1000
starringActorNum = df["starring"] > 3
df["title"] = df["title"].astype("str")
titleLen = df["title"].str.len() <= 10

I want to create a new csv file using the criteria above, but am unsure how to do that as well as how to combine all those attributes into one csv.

Anyone have any ideas?

You want to have your data frame filtered with all of the constraints you present there being applied in parallel? — languitar
– languitar, Commented Feb 11, 2017 at 20:30
Yes. I have a .csv dataset I've loaded in, but want to filter out some of the content based on certain criteria (hence, the examples above). I just don't know how to translate all my constraints to a csv — natalien
– natalien, Commented Feb 11, 2017 at 20:33

unutbu · Accepted Answer · 2017-02-11 20:37:06Z

5

Combine the boolean masks using & (bitwise-and):

mask = viewNum & starringActorNum & titleLen

Select the rows of df where mask is True:

df_filtered = df.loc[mask]

Write the DataFrame to a csv:

df_filtered.to_csv('movies-filtered.csv')

import pandas as pd

df = pd.read_csv("movies.csv")

viewNum = df["views"] >= 1000
starringActorNum = df["starring"] > 3
df["title"] = df["title"].astype("str")
titleLen = df["title"].str.len() <= 10

mask = viewNum & starringActorNum & titleLen
df_filtered = df.loc[mask]

df_filtered.to_csv('movies-filtered.csv')

answered Feb 11, 2017 at 20:37

unutbu

886k197 gold badges1.9k silver badges1.7k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Stephen Rauch · Accepted Answer · 2017-02-11 21:03:11Z

0

You can use the panda.DataFrame.query() interface. It allows text string queries, and is very fast for large data sets.

Something like this should work:

import pandas as pd
df = pd.read_csv("movies.csv")

# the len() method is not available to query, so pre-calculate
title_len = df["title"].str.len()

# build the data frame and send to csv file, title_len is a local variable
df.query('views >= 1000 and starring > 3 and @title_len <= 10').to_csv(...)

edited Feb 11, 2017 at 21:03

answered Feb 11, 2017 at 20:37

Stephen Rauch♦

50.1k32 gold badges118 silver badges143 bronze badges

Collectives™ on Stack Overflow

Writing to a csv using pandas with filters

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related