Setting elasticsearch properties in spark-submit

Question

I'm trying to launch Spark jobs that use Elastic Search input via command line using spark-submit as described in http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/spark.html

I'm setting the properties in a file, but when launching spark-submit it gives the following warnings:

~/spark-1.0.1-bin-hadoop1/bin/spark-submit --class Main --properties-file spark.conf SparkES.jar

Warning: Ignoring non-spark config property: es.resource=myresource
Warning: Ignoring non-spark config property: es.nodes=mynode
Warning: Ignoring non-spark config property: es.query=myquery
...
Exception in thread "main" org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed

My config file looks like (with correct values):

es.nodes      nodeip:port
es.resource   index/type
es.query      query

Setting the properties in the Configuration object in the code works, but I need to avoid this workaround.

Is there a way to set those properties via command line?

can you add your spark.conf file to the question?

John Petrone
– John Petrone

2014-08-12 20:09:48 +00:00
Commented Aug 12, 2014 at 20:09 — John Petrone
– John Petrone, Commented Aug 12, 2014 at 20:09

Ram Ghadiyaram · Accepted Answer · 2016-09-20 12:43:21Z

4

I don't know if you resolved your issue (if so, how?), but I found this solution:

import org.elasticsearch.spark.rdd.EsSpark

EsSpark.saveToEs(rdd, "spark/docs", Map("es.nodes" -> "10.0.5.151"))

Bye

edited Sep 20, 2016 at 12:43

Ram Ghadiyaram

29.4k16 gold badges102 silver badges133 bronze badges

answered Feb 2, 2015 at 13:12

alfheim

3504 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Metropolis · Accepted Answer · 2015-10-26 17:32:53Z

0

When you pass a config file to spark-submit, it only loads configs that start with 'spark.'

So, in my config I simply use

spark.es.nodes <es-ip>

and in the code itself I have to do

val conf = new SparkConf()
conf.set("es.nodes", conf.get("spark.es.nodes"))

edited Oct 26, 2015 at 17:32

answered Mar 23, 2015 at 16:47

Metropolis

2,1282 gold badges20 silver badges39 bronze badges

Collectives™ on Stack Overflow

Setting elasticsearch properties in spark-submit

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related