Python script to skip specific column in CSV files

Question

I have a Python code which filters the data according to specific column and creates multiple CSV files.

Here is my main csv file:

Name,    City,      Email
john     cty_1      [email protected]
jack     cty_1      [email protected]
...
Ross     cty_2      [email protected]
Rachel   cty_2      [email protected]
...

My python logic currently creates separate csv for separate city. Existing python logic is:

from itertools import groupby
import csv

with open('filtered_final.csv') as csv_file:
    reader = csv.reader(csv_file)
    next(reader) #skip header
    
    #Group by column (city)
    lst = sorted(reader, key=lambda x : x[1])
    groups = groupby(lst, key=lambda x : x[1])

    #Write file for each city
    for k,g in groups:
        filename = k[21:] + '.csv'
        with open(filename, 'w', newline='') as fout:
            csv_output = csv.writer(fout)

            csv_output.writerow(["Name","City","Email"])  #header
            for line in g:
                csv_output.writerow(line)

Now, I want to remove the "City" Column on each new CSV files.

Quang Hoang · Accepted Answer · 2020-10-06 03:43:14Z

1

If you data is small enough to put on ram, you can just read the whole thing in and do a groupby:

import pandas as pd

df = pd.read_csv('filtered_final.csv')

for city, data in df[['Name','Email']].groupby(df['City']):
    data.to_csv(f'{city}_data.csv', index=False)

edited Oct 6, 2020 at 3:43

answered Oct 6, 2020 at 3:16

Quang Hoang

151k11 gold badges64 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Dipendra Dangal Over a year ago

Hi Quang, Here you have used 'pd'. Is it a library? Do we need to import that one?

Quang Hoang Over a year ago

@DeependraDangal it's standard shorthand for pandas, which you tagged.

Dipendra Dangal Over a year ago

Okay @Quang . It worked. But in the new CSV files, one additional identifier is automatically added in the first column(before Name and Email column)

Quang Hoang Over a year ago

Oh, I forgot to pass index=False into to_csv. See updates.

avarice · Accepted Answer · 2020-10-06 03:09:57Z

1

Then try to import like:

df = pd.read_csv('filtered_final.csv', usecols=['Name','Email'])

answered Oct 6, 2020 at 3:09

avarice

15.6k3 gold badges19 silver badges38 bronze badges

Collectives™ on Stack Overflow

Python script to skip specific column in CSV files

2 Answers 2

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related