How to read HDFS file from Scala code

Question

I am new to Scala and HDFS:

I am just wondering I am able to read local file from Scala code but how to read from HDFS:

import scala.io.source
object ReadLine {
  def main(args:Array[String]) {
    if (args.length>0) {
      for (line <- Source.fromLine(args(0)).getLine())
        println(line)
      }
    }

in Argument I have passed hdfs://localhost:9000/usr/local/log_data/file1.. But its giving FileNotFoundException error I am definitely missing something.. can anyone help me out here ?

Anahcolus · Accepted Answer · 2017-07-24 02:27:25Z

10

scala.io.source api cannot read from HDFS. Source is used to read from local file system.

Spark

If you want to read from hdfs then I would recommend to use spark where you would have to use sparkContext.

val lines = sc.textFile(args(0))  //args(0) should be hdfs:///usr/local/log_data/file1

No Spark

If you don't want to use spark then you should go with BufferedReader or StreamReader or hadoop filesystem api. for example

val hdfs = FileSystem.get(new URI("hdfs://yourUrl:port/"), new Configuration()) 
val path = new Path("/path/to/file/")
val stream = hdfs.open(path)
def readLines = Stream.cons(stream.readLine, Stream.continually( stream.readLine))

answered Jul 24, 2017 at 2:27

Anahcolus

42.1k6 gold badges75 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to read HDFS file from Scala code

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related