Creating a dataframe from JSON data using PYSPARK

Question

I am trying to create a dataframe out of json data using pyspark module ,but not able to do,tried doing it with sqlContext.read.json but not getting proper result.

sample json data:

{
"userId":"rirani",
"jobTitleName":"Developer",
"firstName":"Romin",
"lastName":"Irani",
"preferredFullName":"Romin Irani",
"employeeCode":"E1",
"region":"CA",
"phoneNumber":"408-1234567",
"emailAddress":"[email protected]"
},
{
"userId":"nirani",
"jobTitleName":"Developer",
"firstName":"Neil",
"lastName":"Irani",
"preferredFullName":"Neil Irani",
"employeeCode":"E2",
"region":"CA",
"phoneNumber":"408-1111111",
"emailAddress":"[email protected]"
}
{
"userId":"thanks",
"jobTitleName":"Program Directory",
"firstName":"Tom",
"lastName":"Hanks",
"preferredFullName":"Tom Hanks",
"employeeCode":"E3",
"region":"CA",
"phoneNumber":"408-2222222",
"emailAddress":"[email protected]"
}

expected o/p:in table format.can anyone help me this.

And what result are you getting? What does o/p:in mean?

botchniaque
– botchniaque

2018-06-22 12:56:11 +00:00
Commented Jun 22, 2018 at 12:56 — botchniaque
– botchniaque, Commented Jun 22, 2018 at 12:56

fatmali · Accepted Answer · 2018-06-22 14:10:11Z

You can do use SparkSession:

my_json = [{ 
     "userId":"rirani",
    "jobTitleName":"Developer", 
    "firstName":"Romin", 
    "lastName":"Irani", 
    "preferredFullName":"Romin Irani",
     "employeeCode":"E1",
     "region":"CA",
     "phoneNumber":"408-1234567",
     "emailAddress":"[email protected]" 
    }, 
    { "userId":"nirani", 
    "jobTitleName":"Developer", 
    "firstName":"Neil", 
    "lastName":"Irani",
    "preferredFullName":"Neil Irani",
    "employeeCode":"E2", "region":"CA",
    "phoneNumber":"408-1111111",
    "emailAddress":"[email protected]" 
    },
    { "userId":"thanks", 
    "jobTitleName":"Program Directory",
    "firstName":"Tom", 
    "lastName":"Hanks", 
    "preferredFullName":"Tom Hanks",         "employeeCode":"E3", "region":"CA", "phoneNumber":"408-2222222",
"emailAddress":"[email protected]"
         }]

json_df = spark.read.json(my_json)
json_df.show()

Collectives™ on Stack Overflow

Creating a dataframe from JSON data using PYSPARK

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related