Update a mutable List within a foreach loop scala spark

Question

I need to update a mutable list with the content of a directory in HDFS, I have the following code witch in spark-shell works but inside an script it doesn't:

import org.apache.hadoop.fs._
import org.apache.spark.deploy.SparkHadoopUtil

var listOfFiles= scala.collection.mutable.ListBuffer[String]()

val hdfs_conf = SparkHadoopUtil.get.newConfiguration(sc.getConf)
    val hdfs = FileSystem.get(hdfs_conf)
    val sourcePath = new Path(filePath)  

 hdfs.globStatus( sourcePath ).foreach{ fileStatus =>
      val filePathName = fileStatus.getPath().toString();
      val fileName = fileStatus.getPath().getName();
      listOfFiles.append(fileName)
  } 

listOfFiles.tail

any help, when running it launches an exception telling that listOfFiles is empty.

What exception do you get when you write it in a scala file? — gonephishing
– gonephishing, Commented Jun 2, 2016 at 13:22
Nothing wrong on scala aspect I guess, maybe check the hdfs.globStatus ( ... ) part again — gonephishing
– gonephishing, Commented Jun 2, 2016 at 13:34
Most probably your hdfc.globalStatus(sourcePath) is not returning anything — Niks
– Niks, Commented Jun 2, 2016 at 13:36
why is it that in spark-shell it works and in an script it doesn't? — Loreto Gómez Arias
– Loreto Gómez Arias, Commented Jun 2, 2016 at 13:36

Przemek · Accepted Answer · 2016-06-02 13:46:01Z

2

You should avoid using mutable collection.

Try:

val listOfFiles = hdfs.globStatus(sourcePath).map{ fileStatus =>
      fileStatus.getPath().getName();
  }

edited Jun 2, 2016 at 13:46

answered Jun 2, 2016 at 13:36

Przemek

2282 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Update a mutable List within a foreach loop scala spark

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related