Spark - ElasticSearch Index creation performance too slow

Question

I am trying to use Apache spark to create an index in Elastic search(Writing huge data to ES).I have done a Scala program to create index using Apache spark.I have to index huge data, which is getting as my product bean in a LinkedList. Then.Then i tried to traverse over the product bean list and create the index. My code given below.

val conf = new SparkConf().setAppName("ESIndex").setMaster("local[*]")
    conf.set("es.index.auto.create", "true").set("es.nodes", "127.0.0.1")
      .set("es.port", "9200")
      .set("es.http.timeout", "5m")
      .set("es.scroll.size", "100")

    val sc = new SparkContext(conf)

    //Return my product bean as a in a linkedList.
    val list: util.LinkedList[product] = getData() 

    for (item <- list) {
      sc.makeRDD(Seq(item)).saveToEs("my_core/json")
    }

The issue with this approach is taking too much time to create the index. Is there any way to create the index in a better way?

Why do you pass data through driver? This is an obvious bottleneck. — zero323
– zero323, Commented Mar 9, 2016 at 9:56
plus if your data fits in memory why to overhead your architecture with spark ? — eliasah
– eliasah, Commented Mar 9, 2016 at 10:00

zero323 · Accepted Answer · 2016-03-09 10:32:15Z

3

Don't pass data through driver unless it is necessary. Depending on what is the source of data returned from getData you should use relevant input method or create your own. If data comes from MongoDB use for example mongo-hadoop, Spark-MongoDB or Drill with JDBC connection. Then use map or similar method to build the required objects and use saveToEs on transformed RDD.
Creating a RDD with as single element doesn't make sense. It doesn't benefit from Spark architecture at all. You just start a potentially huge number of tasks which have nothing with only a single active executor.

edited Mar 9, 2016 at 10:32

answered Mar 9, 2016 at 10:24

zero323

331k108 gold badges982 silver badges958 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Juhan Over a year ago

so is there way to process the entire list than a single object through driver?

zero323 Over a year ago

Once again. Don't use driver to pass or process the data. If you do it, and process data locally there is no reason to use Spark.

Collectives™ on Stack Overflow

Spark - ElasticSearch Index creation performance too slow

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related