How to optimize this Scala code?

Question

I'm learning Scala, curious how to optimize this code. What I have is an RDD loaded from Spark. It's a tab delimited dataset. I want to combine the first column with the second column, and append it as a new column to the end of the dataset, with a "-" separating the two.

For example: column1\tcolumn2\tcolumn3

becomes

column1\tcolumn2\tcolumn3\tcolumn1-column2

val f = sc.textFile("path/to/dataset")

f.map(line => if (line.split("\t").length > 1) 
        line.split("\t") :+ line.split("\t")(0)+"-"+line.split("\t")(1) 
    else 
        Array[String]()).map(a => a.mkString("\t")
 )
 .saveAsTextFile("output/path")

I'd start by doing that string split only once.

maasg
– maasg

2015-05-18 21:39:56 +00:00
Commented May 18, 2015 at 21:39 — maasg
– maasg, Commented May 18, 2015 at 21:39

David Griffin · Accepted Answer · 2015-05-18 23:36:16Z

1

Try:

f.map{ line =>
  val cols = line.split("\t")
  if (cols.length > 1) line + "\t" + cols(0) + "-" + cols(1)
  else line
}

edited May 18, 2015 at 23:36

answered May 18, 2015 at 21:57

David Griffin

14k5 gold badges49 silver badges66 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Sean Bollin Over a year ago

Thanks, yeah change it from map{ to map( allowed me to set a val inside.

Collectives™ on Stack Overflow

How to optimize this Scala code?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related