0

I am a newbie to scala (and functional programming, basically). I am trying to loop over a list of rows (can think as strings), where each string will be passed to a different scala method, where I am doing some manipulation on the input string and then return a string to the for-loop.

below is not the working code, but this is what I am expecting to work. 

     val input_list = spark.read
          .format("com.crealytics.spark.excel")
          .option("sheetName" , "SchemaInfo")
          .option("useHeader", "true")
          .schema(profilerSchema)
          .load(path)  // this is spark dataframe, which has rows.

 val columnNames : List[String] = new List("Hello" , "world");

         var outputList = new ListBuffer[String]();

        // Here i am iterating the input_list where i pass each ele to getString() method where  
        // it returns the final string which i want to add to outputList. 

         input_list.foreach(i => {
                val res: String = getString(row, columnNames)
                 outputList += res;
               }));


         def getString(row: Row, schemaNames: List[String]) : String  = {

        // some implementation where it returns a string.
        }

Below is the error message I am getting (discard the line number. its getting at the foreach loop.).

Error:(57, 14) overloaded method value foreach with alternatives:
  (func: org.apache.spark.api.java.function.ForeachFunction[org.apache.spark.sql.Row])Unit <and>
  (f: org.apache.spark.sql.Row => Unit)Unit
 cannot be applied to (org.apache.spark.sql.Row => scala.collection.mutable.ListBuffer[String])
    excel_df.foreach{row => (jsonStrList += convertRowToJSON(row, columnNames))};

I am having a hard time writing the logic. any help is really appreciated.

Improve this question
4
  • Can you clarify what isn't working right now? What error messages are you getting? Or if you're not getting any, what's the output you see versus what you expect? Also, getString asks for a Row but you pass it a String, I assume it should take String in this example? Commented Nov 1, 2019 at 17:55
  • HI Ethan, my bad. i had updated the question with the error message. thanks Commented Nov 1, 2019 at 18:00
  • We don't recommend use of var in Scala. Especially if you are trying to do functional programming. There are no mutations in FP. Please use a For/yield or it seems like even a .map would suffice in this case Commented Nov 1, 2019 at 18:07
  • 1
    You have an extra ending parentheses in the expression where you call foreach. However, I don't think that's why your code is failing. Your real problem is that the input variable to your anonymous function, i, is never used in your function. You are either omitting some important code, or else something else is going on. For starters, can you tell us the type for input_list? It would help us debug your issue. Commented Nov 1, 2019 at 18:12

1 Answer 1

Reset to default
3 
input_list.foreach(i => {
      val res: String = getString(row, columnNames)
       outputList += res;
     });

Your function inside the foreach method returns the outputList value. If you look at the signature, the return value should be Unit - which means the method doesn't return a value. It assumes you do some calculation that doesn't return anything.

You should use map instead of foreach. You don't need foreach and the outputList variable.

input_list.map(row => {
       // Logic to return the item that you want to compute
     })
     .toList()

You have to shift your way of thinking from an imperative style where you declare a collection and then loop through the items of another collection and add the result of your calculations to the first collection to a functional style that uses the map/filter methods.

Example:

List(1,2,3,4,5,6,7,8,9,10)
// filter gives you the list of even numbers between 1 and 10
.filter(i => i % 2 == 0) 
// This gives you the squares of the even numbers between 1 and 10
.map(i => i * i)
// This gives the doubles of the squares of the even numbers
.map(i => i * 2)

You can also do this:

val evenNumbers = List(1,2,3,4,5,6,7,8,9,10)
// filter gives you the list of even numbers between 1 and 10
.filter(i => i % 2 == 0) 

val squares= evenNumbers
// This gives you the squares of the even numbers between 1 and 10
.map(i => i * i)

val doubleSquares = squares
// This gives the doubles of the squares of the even numbers
.map(i => i * 2)

// this will return a tuple with lists when it's the last statement in a function or method.
(squares, doubleSquares)

As you can see there are no ListBuffer objects declared. The example is a bit contrived, you could combine the last two maps into one but I wanted to prove a point. You can do other operations, you can group by items, sort them and so on.

You need to read more about scala and FP in general. This is a good introduction: https://docs.scala-lang.org/overviews/scala-book/introduction.html and this: https://docs.scala-lang.org/overviews/scala-book/passing-functions-around.html.

You can also trying things here inside the browser: https://scastie.scala-lang.org.

Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

I think it's worth mentioning that in this case, using vars and mutable collections is a straight up error, since the code is being run with spark.
In case costa's solution isn't clear enough, here is what the code should look like: val outputList = input_list.map(row => getString(row, columnNames)).toList(). Something like that. I'm not sure if this code snippet compiles because I don't know the type of input_list, but oh well. outputList will be a List[String].
thanks @Allen. it working. as I am working on a dataframe, I need to use the collect() method and then make it to list by using toList. val jsonStrList : List[String] = excel_df.map(row => convertRowToJSON(row, columnNames)).collect().toList
@Costa, what do you meant by, I don't need an outputList variable ? As from the inputList, once I get the output, I need to save it to a list right? Am I thinking wrong ?
@marc: In the FP style of programming you don't do that. You could store your temporary results in variables declared with val (not var). Think of the whole thing as a pipeline. You start with some initial list, you apply it transformations(i.e functions) and you get another list as a result, then you transform that list if you need to and so on. Scala is very good at retaining the original collection type, i.e. if you start with a List and you apply filter and map you end up with a list.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.