Pandas DataFrame.to_csv() doesn't output all columns

Question

I have a large dataframe with ~450 columns. I need to export it to csv, and for that I use the next command:

df.to_csv('output.csv', sep='\t', header=False, index=False, na_rep='null')

The resulting file is missing several of the trailing columns. I tried to update pandas display options, but that didn't solve the problem:

pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', -1)
pd.set_option('display.width', None)

Any help is appreciated.

Background: I need to export data from one DB table to the other, adding one more column on the fly. I export data to csv, create a dataframe from it, add a column, and then need to export it to csv again to import into the resulting table.

Why can't you copy the dataframe and a column to the new df? — Rahul Agarwal
– Rahul Agarwal, Commented Dec 6, 2018 at 8:31
Have you tried setting <header=True> to check which columns are missing? alternatively, try using <sep=','> and see if issue persists. — ParvBanks
– ParvBanks, Commented Dec 6, 2018 at 8:32
Maybe it goes wrong when you're importing the file from DB to a df? Or do you have the right amount of columns when you do the import and it only goes wrong with the final export? — Sander van den Oord
– Sander van den Oord, Commented Dec 6, 2018 at 8:49

NothisIm · Accepted Answer · 2019-04-15 14:54:43Z

1

As it turned out, the original data had a quotation mark somewhere. This answer to another question helped me sort it out. I resolved the problem with:

pd.read_csv(file_obj, sep='\t', quoting=csv.QUOTE_NONE)

answered Apr 15, 2019 at 14:54

NothisIm

631 silver badge7 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pandas DataFrame.to_csv() doesn't output all columns

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related