How to ignore empty columns in a dataframe?(Pandas)

Question

I want to ignore empty columns in a dataframe.

For Example:

sample.csv

Id  Name  Address    Contact   Item   Rate Qty  Price
1   Mark  California 98429102  Shirt  57    2    8
2   Andre Michigan   92010211

I have tried:

import pandas as pd
df = pd.read_csv('sample.csv')
df = df.fillna('')

df.to_csv('sample.txt',sep='*',index=False, header=False)

The sample.txt looks like

1*Mark*California*98429102*Shirt*57*2*8
2*Andre*Michigan*92010211****

I want to remove the empty columns here. The sample.txt should look like this:

1*Mark*California*98429102*Shirt*57*2*8
2*Andre*Michigan*92010211

That looks like a badly specified format. I don't think pandas will help you here. You'll probably have to do some post-processing on the text file directly. — IanS
– IanS, Commented Feb 12, 2021 at 10:02
At best you could split the dataframe in 2 (with a boolean mask like df['item'].isna()) and write 2 separate CSV files. — IanS
– IanS, Commented Feb 12, 2021 at 10:04

Rob Raymond · Accepted Answer · 2021-02-12 10:53:52Z

2

Just use a memory buffer and strip()

import io
df = pd.read_csv(io.StringIO("""1*Mark*California*98429102*Shirt*57*2*8
2*Andre*Michigan*92010211****"""), sep="*", header=None)

with open("sample.csv", "w") as f: 
    f.write("\n".join([l.strip("*") for l in df.to_csv(sep="*",header=None, index=None).split("\n")]))

with open("sample.csv") as f: print(f.read())

output

1*Mark*California*98429102*Shirt*57.0*2.0*8.0
2*Andre*Michigan*92010211

answered Feb 12, 2021 at 10:53

Rob Raymond

31.5k3 gold badges19 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Jaroslav Bezděk · Accepted Answer · 2022-02-10 09:37:17Z

1

What about

sep = '*'
(
    df
    .applymap(str)
    .apply(
        # Removes all empty fields
        # axis=1, func=lambda s: sep.join(el for el in s if el)
        # Removes trailing fields
        axis=1, func=lambda s: sep.join(s).strip('*')
    )
    .to_csv('sample.txt', index=False, header=False)
)

edited Feb 10, 2022 at 9:37

Jaroslav Bezděk

7,7156 gold badges34 silver badges59 bronze badges

answered Feb 12, 2021 at 10:32

keepAlive

6,7055 gold badges29 silver badges43 bronze badges

3 Comments

Rikky Bhai Over a year ago

This removes empty elements from all the fields. I just want to remove trailing fields.

keepAlive Over a year ago

@Atom Indeed. See update. Rather minor. Any question ?

keepAlive Over a year ago

Ok @Jaroslav. As long as it derives from K&R's style (or 1TBS), I am okay with it.

Collectives™ on Stack Overflow

How to ignore empty columns in a dataframe?(Pandas)

2 Answers 2

output

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

output

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related