elasticsearch-pyspark : not getting specific fields from document (getting all fields) even after specifying with spark

Question

I'm trying to extract some data from elasticsearch with pyspark. I want to extract only few fields (not all) from the documents. So, I'm making a post request from the software "Postman" (for testing purpose) with following url and body.It's giving perfect output as expected. But when I'm using same body with spark code, it's extracting all the fields from the specified documents which is not desired. Can anyone tell what might be the reason for such weird behavior ? Thanks in advance !

Spark version 2.3, Elasticsearch version 6.2, postman body type = application/json

This is what I'm doing with postman :

`url : localhost:9200/test-index4/school/_search`

`body : 
{
    "query":
     {
         "ids":
           {
               "values":["8","9","10"]
           }
     },
     "_source":
     {
         "includes":["name"]
     }
}`

Below is what I'm doing with pyspark :

`body = "{"query":{"ids":{"values":["8","9","10"]}},"_source":{"includes":["name"]}}"
df = self.__sql_context.read.format("org.elasticsearch.spark.sql") \
            .option("es.nodes", "localhost") \
            .option("es.port", "9200") \
            .option("es.query", body) \
            .option("es.resource", "test-index4/school") \
            .option("es.read.metadata", "true") \
            .option("es.read.metadata.version", "true") \
            .option("es.read.field.as.array.include", "true") \
            .load()

`

This might be what you need. The es.read.field.include config. — mkaran
– mkaran, Commented May 10, 2018 at 9:38
It worked ! Thanks a lot @mkaran. But still do you know reason for such weired behavior ? Also es.read.metadata.version doesn't read version. Can you please explain why ? — amol_shaligram
– amol_shaligram, Commented May 10, 2018 at 9:43
I'm glad it worked! Well, by default, elasticsearch will bring you all the fields, from the docs: if es.read.field.include is not set, then it defaults to null and all fields are returned. As for the metadata.version I don't know, since you've set both es.read.metadata and es.read.metadata.version to true which is the correct configuration from what I can tell. — mkaran
– mkaran, Commented May 10, 2018 at 9:53

priyanshi · Accepted Answer · 2018-05-10 10:26:40Z

4

Try setting es.read.field.include in config with value as comma seperated field list. e.g. "es.read.field.include","field1,field2,..."

answered May 10, 2018 at 10:26

priyanshi

816 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

elasticsearch-pyspark : not getting specific fields from document (getting all fields) even after specifying with spark

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related