java.lang.IllegalArgumentException: java.net.UnknownHostException: tmp

Question

I work an application Spark-scala and I built project with sbt, my arborescence is: projectFilms/src/main/scala/AppFilms I have 3 files in HDFS, these directories is: hdfs/tmp/projetFilms/<my_3_Files>, When I run my code by this command line "sbt run", it genere an error:

java.lang.IllegalArgumentException: java.net.UnknownHostException: tmp

and this:

  [trace] Stack trace suppressed: run last compile:run for the full output.
 ERROR Utils: uncaught error in thread SparkListenerBus, stopping SparkContext
java.lang.InterruptedException

This is my code:

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.rdd._   

object appFilms {

val conf = new SparkConf().setAppName("system of recommandation").setMaster("local[*]")
val sc = new SparkContext(conf)
def main(args: Array[String]) {

val files = sc.wholeTextFiles("hdfs://tmp/ProjetFilm/*.dat")
//val nbfiles = files.count
println("Hello my application!")
sc.stop()
}
}

I can't read my files from hdfs, but when i write

root@sandbox projectFilms# hadoop fs -cat /tmp/ProjetFilms/*

How I can read the content of all my files from HDFS, knowing that I work always by the same command.

Please can you answers me!

Nirmal Ram · Accepted Answer · 2016-11-28 10:37:48Z

5

the error IllegalArgumentException: java.net.UnknownHostException: tmp is because in wholeTextFiles value its taking tmp as the hostname. Replace value with hdfs:///tmp/ProjetFilm/*.dat

answered Nov 28, 2016 at 10:37

Nirmal Ram

1,2202 gold badges10 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

sirine Over a year ago

Thank you for your answers!

sirine Over a year ago

I wrote the complete path name of the HDFS files follows: val files = sc.wholeTextFiles("hdfs://sandbox.hortonworks.com:8020/tmp/ProjetFilm/*.dat")

T. Gawęda · Accepted Answer · 2016-11-28 10:42:36Z

2

Use:

val files = sc.wholeTextFiles("hdfs:///tmp/ProjetFilm/*.dat")

There is one additional / after hdfs://, which is a protocol name. You must go to /tmp/... via hdfs:// protocol, that's why URL needs additional /. Without this, Spark is trying to reach host tmp, not folder

edited Nov 28, 2016 at 10:42

answered Nov 28, 2016 at 10:34

T. Gawęda

16.1k5 gold badges51 silver badges62 bronze badges

Collectives™ on Stack Overflow

java.lang.IllegalArgumentException: java.net.UnknownHostException: tmp

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related