Json file : create table or dataframe by using python pyspark?

Question

I have a file that have this type of schema :

and trying to create a table or dataframe base on them. Before that, have managed to upload the file using this script :

first_row_is_header = "true"
infer_schema = "true"
delimiter = ":"
df4 = spark.read.option("multiLine","true") \
.option("inferSchema", infer_schema) \
.option("sep", delimiter) \
.json ("/FileStore/tables/Ca_sect2.json")

End result become like this :

How do I convert the result below into table form? Have tried several types of scripts like dataFrame.createOrReplaceTempView, but keep on failing to get the result. Grateful for some help, as have been trying for 2 days.

Thank you very much.

blackbishop · Accepted Answer · 2019-12-29 13:33:46Z

1

What you want is to explode the array structure of developerSales:

df.withColumn("sales", explode(col("developerSales")))\
  .select(col("developer"), col("sales.*"))\
  .show()

answered Dec 29, 2019 at 13:33

blackbishop

32.8k11 gold badges61 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Json file : create table or dataframe by using python pyspark?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related