How to read each Json object in my input file to a row in spark DataFrame

Question

I am new to spark and Scala and I am trying to learn spark for one of my learning project. I have a JSON file which look like this:

[
  {
"year": 2012,
"month": 8,
"title": "Batman"
},
  {
"year": 2012,
"month": 8,
"title": "Hero"
 },
 {
"year": 2012,
"month": 7,
"title": "Robot"
 }
]

I started reading this json to spark DataFrame file so i tried following:

spark.read
  .option("multiline", true)
  .option("mode", "PERMISSIVE")
  .option("inferSchema", true)
  .json(filePath)

It reads the JSON but convert the data to spark columns. My requirement is to read each data object as one individual column.

I want to read it to a spark DataFrame where I expect output like following:

+----------------------------------------+
|json                                    |
+----------------------------------------+
|{"year":2012,"month":8,"title":"Batman"}|
|{"year":2012,"month":8,"title":"Hero"}  |
|{"year":2012,"month":7,"title":"Robot"} |
|{"year":2011,"month":7,"title":"Git"}   |
+----------------------------------------+

Kishore · Accepted Answer · 2018-12-28 12:29:21Z

1

Use toJSON

val df = spark.read
  .option("multiline", true)
  .option("mode", "PERMISSIVE")
  .option("inferSchema", true)
  .json(filePath).toJSON

Now

df.show(false)

+----------------------------------------+
|value                                   |
+----------------------------------------+
|{"month":8,"title":"Batman","year":2012}|
|{"month":8,"title":"Hero","year":2012}  |
|{"month":7,"title":"Robot","year":2012} |
+----------------------------------------+

edited Dec 28, 2018 at 12:29

answered Dec 28, 2018 at 11:20

Kishore

5,9215 gold badges32 silver badges54 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Dharm Over a year ago

Thanks for the quick response. My purpose is not to print it but read in such a way that each JSON object becomes a row. I guess reading first and then converting to JSON will have unnecessary cost. is there any better way to achieve this ?

Rohit Nimmala Over a year ago

Dharm Over a year ago

I want the way Kishore has mentioned but if you look at the code it is first reading (.json) and converting it to JSON (.toJSON) which I think is a costly way of doing it. So I want to know if this is the most efficient way or not.

Collectives™ on Stack Overflow

How to read each Json object in my input file to a row in spark DataFrame

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related