Scala Convert For Loop to Functional Method

Question

I would like to convert the following for loop into a functional Scala method.

for (i <- 15 to 25){
  count_table_rdd = count_table_rdd.union(training_data.map(line => (i+"_"+line(i)+"_"+line(0), 1)).reduceByKey(_ + _))
}

I have tried look at the foreach method, but I do not want to transform every item, just 15 through 25.

Scala collections have a slice(from:Int, to:Int) method on them, so if you could slice and then foreach you could be all set — cmbaxter
– cmbaxter, Commented Apr 10, 2015 at 18:02
Do you really need the value of i in your actual use case? Or just line(i)? — The Archetypal Paul
– The Archetypal Paul, Commented Apr 10, 2015 at 21:13

Rex Kerr · Accepted Answer · 2015-04-10 18:02:10Z

3

You can fold.

val result = (count_table_rdd /: (15 to 25)){ (c, i) => c.union(...) }

If you see that you've got a set of data and you're pushing a value through it doing updates to that value, you should reach for a fold because that's exactly what it does.

answered Apr 10, 2015 at 18:02

Rex Kerr

168k27 gold badges325 silver badges411 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Justin Pihony Over a year ago

I thought the domino operator is frowned upon now?

Rex Kerr Over a year ago

@JustinPihony - Eh, Martin Odersky makes a good case that it very accurately visually represents what's going on. And foldLeft makes you switch the order of arguments in your head between the initial list/parameter and the closure.

curious · Accepted Answer · 2015-04-10 19:19:44Z

1

You may use tailrec too but @rex's method is what you should be following. It will not compile, specify Type of your count_table_rdd and res accordingly

tailrec version :

@annotation.tailrec
  def f(start: Int = 15, end: Int = 25,res:List[Your_count_table_rdd_Type]=Nil): List[Your_count_table_rdd_Type] = {
    if (start > end) count_table_rdd
    else {
     val temp = res ++ training_data.map(line => (start + "_" + line(start) + "_" + line(0), 1)).reduceByKey(_ + _)
      f(start + 1, end,temp)
    }
  }

  f()

you can specify start and end too.

f(30,45)

edited Apr 10, 2015 at 19:19

answered Apr 10, 2015 at 18:05

curious

2,92817 silver badges25 bronze badges

1 Comment

Rex Kerr Over a year ago

That's not functional; you're mutating count_table_rdd. Try again?

maasg · Accepted Answer · 2015-04-10 19:42:56Z

1

Taking this from the Spark perspective, it could be better to do this by transforming the trainingDataRDD instead of looping to select given columns.

Something like:

trainingData.flatMap(line => (15 to 25).map(i => (i+"_"+line(i)+"_"+line(0), 1)))
        .reduceByKey(_ + _)

This will be more efficient that joining pieces of an RDD together using union.

answered Apr 10, 2015 at 19:42

maasg

37.5k14 gold badges91 silver badges116 bronze badges

Collectives™ on Stack Overflow

Scala Convert For Loop to Functional Method

3 Answers 3

2 Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related