Matching Column name from Csv file in spark scala

Question

I want to take headers (column name) from my csv file and the want to match with it my existing header. I am using below code:

val cc = sparksession.read.csv(filepath).take(1)

Its giving me value like:

Array([id,name,salary])

and I have created one more static schema, which is giving me value like this:

val ss=Array("id","name","salary")

and then I'm trying to compare column name using if condition:

if(cc==ss){
  println("matched")
} else{
  println("not matched")
}

I guess due to [] and () mismatch its always going to else part is there any other way to compare these value without considering [] and ()?

Do you only want to compare column names or their values as well? — code.gsoni
– code.gsoni, Commented Apr 22, 2020 at 10:25

Shaido · Accepted Answer · 2020-04-22 09:18:08Z

4

First, for convenience, set the header option to true when reading the file:

val df = sparksession.read.option("header", true).csv(filepath)

Get the column names and define the expected column names:

val cc = df.columns
val ss = Array("id", "name", "salary")

To check if the two match (not considering the ordering):

if (cc.toSet == ss.toSet) {
  println("matched")
} else {
  println("not matched")
}

If the order is relevant, then the condition can be done as follows (you can't use Array here but Seq works):

cc.toSeq == ss.toSeq

or you a deep array comparison:

cc.deep == d.deep

answered Apr 22, 2020 at 9:18

Shaido

28.6k26 gold badges76 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

ZBe · Accepted Answer · 2020-04-22 09:33:52Z

0

First of all, I think you are trying to compare a Array[org.apache.spark.sql.Row] with an Array[String]. I believe you should change how you load the headers to something like: val cc = spark.read.format("csv").option("header", "true").load(fileName).columns.toArray. Then you could compare using cc.deep == ss.deep.

answered Apr 22, 2020 at 9:33

ZBe

231 silver badge3 bronze badges

1 Comment

Up Ap Over a year ago

val cc= spark.read.csv("filepath").take(1)(0).toString and then cretaed ss as ("","")after that applied if condition. Will try your solution also. Thanks!!

Shaido · Accepted Answer · 2020-04-27 09:10:39Z

0

Below code worked for me.

val cc= spark.read.csv("filepath").take(1)(0).toString

The above code gave output as String:[id,name,salary].

created one one stating schema as

val ss="[id,name,salary]"

then wrote the if else Conditions.

edited Apr 27, 2020 at 9:10

Shaido

28.6k26 gold badges76 silver badges82 bronze badges

answered Apr 22, 2020 at 10:10

Up Ap

1074 silver badges14 bronze badges

Collectives™ on Stack Overflow

Matching Column name from Csv file in spark scala

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related