reading a file into array list in scala spark

Question

I am completely new to spark and scala.

I want to read a file into an array list.

This is how its done in java.

List<String> sourceRecords;
sourceRecords = new ArrayList<String>();
BufferedReader SW;
SW = new BufferedReader(new FileReader(srcpath[0].toString()));
String srcline ;
while ((srcline = SW.readLine()) != null)  {
sourceRecords.add(srcline.toString());
}

How to do it in scala in spark

zsxwing · Accepted Answer · 2014-03-10 07:26:42Z

5

It's very easy. For example,

val rdd = sc.textFile("your_file_path")
val sourceRecords = rdd.toArray

However, you don't need to convert rdd to Array. You can manipulate rdd like an Array.

You can find more information in https://spark.incubator.apache.org/examples.html

answered Mar 10, 2014 at 7:26

zsxwing

21k4 gold badges41 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

zsxwing Over a year ago

Docs for Scala, or for Spark?

zsxwing Over a year ago

For Scala, I recommend this book: amazon.com/Programming-Scala-Comprehensive-Step-Step/dp/… . For Spark, I recommend this book : packtpub.com/fast-data-processing-with-spark/book

Pooja3101 Over a year ago

Thanx. I need to convert a map reduce code into scala. I have seen the map function in scala. But i dont understand it. like in wordcount val file = spark.textFile("hdfs://...") val counts = file.flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey(_ + _) counts.saveAsTextFile("hdfs://...") But my mapper in mapreduce has a complex algorithm. how can i implement it in scala.

zsxwing Over a year ago

If you have no time to learn Scala, you can use Spark Java APIs.

Pooja3101 Over a year ago

Can u tell how does the map function for spark java work. like in hadoop map function is called once for each line of file. is it the same way it works in spark

|

Collectives™ on Stack Overflow

reading a file into array list in scala spark

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related