I've tried:
val textLines = spark.sparkContext.textFile("file.txt")
val rowRDD = Row.fromSeq(textLines.map(_.split(",")))
However, I'm getting the error
type mismatch;
found : org.apache.spark.rdd.RDD[Array[String]]
required: Seq[Any]
How can i fix the map?
If what you're trying to do is to load a CSV into a DataFrame, there's a simpler way:
val dataframe: DataFrame = spark.read.csv("file.text")
Alternatively, if you're truly interested in converting an RDD[String] into an RDD[Row] - here's how you'd do that:
val rows: RDD[Row] = textLines.map(_.split(",")).map {
a: Array[_] => Row(a: _*)
}
But note that this might create "uneven" rows (if your data isn't a proper CSV, different rows might have different number of columns, which would make this RDD pretty unusable).
Map is a Higher Order Function function. The return value is always wrapped the the type on which you are applying map function
Example :- List("Answer","Question","StackOverFlow").map(x => x.length)
Then this map function applied on String value but returns the Int Value but this is wrapped on List i.e. List[String] => List[Int]
Here Your map is applied on textLines which is of Type RDD[String]
Then you have applied map over RDD[String] that return RDD[Array[String]]
not Array[String]
You just need to use collect() method like;
val textLines = spark.sparkContext.textFile("file.txt")
val wordsArray = textLines.flatMap(_.split(",")).collect
val rowRDD = Row.fromSeq(wordsArray)
I am using collect() method that will returns you Array[String] and i am using flatMap instead of map because flatMap Flattens the value
otherwise on collect() you will get Array[Array[String]]
you can use map if it fits your scenario.
Hope this solve your problem
