spark scala dataframe adding 1 to all the values in a column

Question

i am new to spark dataframe .i have a text file having data like

schoolid,classid,studentid,subject,marks
bjs,5,111,hindi,23
bjs,5,222,maths,78
bjs,7,333,bio,89
bjs,1,444,chemistry,67
ghs,2,555,bio,78
ghs,2,666,phy,56
ghs,9,777,drawing,56

i want to convert this data to dataframe and add 1 to each of the values under marks column

so code I am using is

val df = sparkSession.read.format("csv").option("header","true").load("samplefile1.txt")
 val newdf = df.select(col($"marks"+1)).show()

but the error that I am getting is

type mismatch; found : org.apache.spark.sql.Column required: String

can I have help with the correct syntax

Shaido · Accepted Answer · 2018-09-10 05:10:40Z

3

Try this solution:

df.withColumn("marks",col("marks") + lit(1)).show

edited Sep 10, 2018 at 5:10

Shaido

28.6k26 gold badges76 silver badges82 bronze badges

answered Sep 9, 2018 at 21:38

Chandan Ray

2,0911 gold badge13 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Binzi Cao · Accepted Answer · 2018-09-10 01:22:28Z

2

 df.withColumn("marks", expr("marks +1").cast("integer")).show

Output :

+--------+-------+---------+---------+-----+
|schoolid|classid|studentid|  subject|marks|
+--------+-------+---------+---------+-----+
|     bjs|      5|      111|    hindi|   24|
|     bjs|      5|      222|    maths|   79|
|     bjs|      7|      333|      bio|   90|
|     bjs|      1|      444|chemistry|   68|
|     ghs|      2|      555|      bio|   79|
|     ghs|      2|      666|      phy|   57|
|     ghs|      9|      777|  drawing|   57|
+--------+-------+---------+---------+-----+

answered Sep 10, 2018 at 1:22

Binzi Cao

1,0855 silver badges14 bronze badges

Collectives™ on Stack Overflow

spark scala dataframe adding 1 to all the values in a column

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related