Add headers in apache spark Dataset in JAVA

Question

The following question contains solution to add headers in dataframe in Scala language. I want to add headers in a Dataset in java language. add header and column to dataframe spark

I am reading a file that does not contain headers:

Dataset<Row> ds= spark.read().format("csv").option("header", "false").load(filepath);

and ds.show() prints this:

+----------+----------------+----------+----+----+---------+----+
|       _c0|             _c1|       _c2| _c3| _c4|      _c5| _c6|
+----------+----------------+----------+----+----+---------+----+
|04/13/2019|             US1|04/13/2019|null|null|      abc|null|
|04/13/2019|             US1|04/13/2019|null|null|    qwert|null|
|04/13/2019|             US1|04/13/2019|null|null|     xyzz|null|
+----------+----------------+----------+----+----+---------+----+

The desired output is with my headers:

+----------+----------------+----------+----+----+---------+----+
| orderDate|          symbol|  sellDate| prc|  id|  product| cod|
+----------+----------------+----------+----+----+---------+----+
|04/13/2019|             US1|04/13/2019|null|null|      abc|null|
|04/13/2019|             US1|04/13/2019|null|null|    qwert|null|
|04/13/2019|             US1|04/13/2019|null|null|     xyzz|null|
+----------+----------------+----------+----+----+---------+----+

Can anyone please help in this regard?

user0204 · Accepted Answer · 2019-05-23 15:16:06Z

1

I have found the answer to my question.

toDF() can be used to add headers as follows:

Dataset<Row> ds= spark.read().format("csv").option("header", "false").load(filepath).toDF("orderDate","symbol","selldate","prc","id","product","cod")

This can also be used to rename headers if they exists. like this:

Dataset<Row> ds= spark.read().format("csv").option("header", "true").load(filepath).toDF("orderDate","symbol","selldate","prc","id","product","cod")

answered May 23, 2019 at 15:16

user0204

2611 gold badge5 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Add headers in apache spark Dataset in JAVA

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related