Adding new row in Scala-Spark dataframe based on a existing row

Question

I have data in Json dataframe like below.

{"nm": 1233, "date": "2017-01-23", "name": [],"id": "9253194"}
{"nm": 1234, "date": "2017-01-23", "name": [],"id": "9253196"}
{"nm": 1235, "date": "2017-01-23", "name": [],"id": "9253195"}

how can add a new row with an indexed column to insert in to elastic search in scala.

{"create": {"_type": "usd", "_id": "92531964", "_index": "amount"}}
{"nm": 1233, "date": "2017-01-23", "name": [],"id": "9253194"}
{"create": {"_type": "usd", "_id": "92531966", "_index": "amount"}}
{"nm": 1234, "date": "2017-01-23", "name": [],"id": "9253196"}
{"create": {"_type": "usd", "_id": "92531965", "_index": "amount"}}
{"nm": 1235, "date": "2017-01-23", "name": [],"id": "9253195"}

here _id I am deriving from existing column and _type, _index are constants.

Joe K · Accepted Answer · 2017-08-02 17:32:27Z

1

Use flatMap:

input.flatMap { x => Seq(x, transform(x)) }

Since these records have different schema, you might have to output them just as strings.

answered Aug 2, 2017 at 17:32

Joe K

18.5k2 gold badges42 silver badges61 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Adding new row in Scala-Spark dataframe based on a existing row

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related