Add column names to data read from csv file without column names

Question

I am using Apache Spark with Scala.

I have a csv file that does not have column names in the first row. It's like this:

28,Martok,49,476
29,Nog,48,364
30,Keiko,50,175
31,Miles,39,161

The columns represent ID, name, age, numOfFriends.

In my Scala object, I am creating dataset using SparkSession from csv file as follows:

val spark = SparkSession.builder.master("local[*]").getOrCreate()
val df = spark.read.option("inferSchema","true").csv("../myfile.csv")
df.printSchema()

When I run the program, the result is:

|-- _c0: integer (nullable = true)
|-- _c1: string (nullable = true)
|-- _c2: integer (nullable = true)
|-- _c3: integer (nullable = true)

How can I add names to the columns in my dataset?

Maybe this helps: stackoverflow.com/questions/40653813/…

mrks
– mrks

2017-11-05 11:09:45 +00:00
Commented Nov 5, 2017 at 11:09 — mrks
– mrks, Commented Nov 5, 2017 at 11:09

Leo C · Accepted Answer · 2017-11-05 12:18:37Z

26

You can use toDF to specify column names when reading the CSV file:

val df = spark.read.option("inferSchema","true").csv("../myfile.csv").toDF(
  "ID", "name", "age", "numOfFriends"
)

Or, if you already have the DataFrame created, you can rename its columns as follows:

val newColNames = Seq("ID", "name", "age", "numOfFriends")
val df2 = df.toDF(newColNames: _*)

edited Nov 5, 2017 at 12:18

answered Nov 5, 2017 at 11:24

Leo C

22.5k3 gold badges28 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user812142 Over a year ago

can you provide a similar solution for java. I have a Dataset<Row> without headers and I want to select some columns not all from it.

Leo C Over a year ago

@user812142, I haven't done much with Java on Spark. Perhaps this SO answer might give some hints; if not, I would suggest posting a separate question with specific requirement and sample data.

pnv Over a year ago

Can you please provide a solution for the problem in PySpark?

padmaja ramesh · Accepted Answer · 2021-03-15 06:08:04Z

1

toDf

method can be used, where you can pass in the column name in spark java.

Example:

Dataset<Row> rowsWithTitle = sparkSession.read().option("header", "true").option("delimiter", "\t").csv("file").toDF("h1", "h2");

answered Mar 15, 2021 at 6:08

padmaja ramesh

113 bronze badges

Collectives™ on Stack Overflow

Add column names to data read from csv file without column names

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related