Save Pyspark Dataframe into csv without headers

Question

I'm looking for a way to save a dataframe into a csv file without specifying headers.

I tried the code below but it did not work.

ratingsDF.coalesce(1).write.option("header", "false")\
.csv("csv_file_without_headers.csv")

For the dataframe below:

+---------+-----+
|   x|   y|    z|
+----+----+-----+
|   0|   a|    5|
|   1|   b|   12|
|   2|   c|    7|
|   3|   d|   27|
|   4|   e|  149|
|   5|   f|   19|
+---------+-----+

The expected result of csv:

O,a,5
1,b,12
2,c,7
3,d,27
4,e,149
5,f,19

notNull · Accepted Answer · 2020-08-15 13:56:51Z

2

Your option looks correct and csv files that is getting written will not be having headers.

In Spark it is not possible to write to file csv_file_without_headers.csv instead check for csv_file_without_headers.csv directory.

In the directory you can see all the files in the directory with out header.

answered Aug 15, 2020 at 13:56

notNull

31.8k4 gold badges41 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Daeho Ro · Accepted Answer · 2020-08-15 13:58:32Z

1

Try this,

ratingsDF.coalesce(1).write.csv("/path/to/save/csv/")

where it will save the csv without header by default. You cannot specify the csv file name but the path only.

answered Aug 15, 2020 at 13:58

Daeho Ro

13.7k4 gold badges25 silver badges50 bronze badges

1 Comment

Mus Over a year ago

Thank you. Even with the code I've put, a directory named csv_file_without_headers.csv was created. And the csv file is inside it

adarsh vemali · Accepted Answer · 2022-05-26 08:27:56Z

0

You can simply do:

ratingsDF.coalesce(1).write.csv("csv_file_without_headers.csv", header = False)

answered May 26, 2022 at 8:27

adarsh vemali

11 bronze badge

Collectives™ on Stack Overflow

Save Pyspark Dataframe into csv without headers

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related