How to create DataFrame Schema from Json schem file

Question

My use case is to read an existing json-schema file, parse this json-schema file and build a Spark DataFrame schema out of it. To start off I followed the steps mentioned here.

Steps followed
1.Imported the library from Maven
2.Restarted the cluster
3.Created a sample JSON schema file
4.Used this code to read the sample schema file
val schema = SchemaConverter.convert("/FileStore/tables/schemaFile.json")

When I run above command I get error: not found: value SchemaConverter

To ensure that the library is being called I reattached the notebook to cluster after restarting the cluster.

In addition to trying out the above method, I tried the below as well. I replaced jsonString with the actual JSON schema.

import org.apache.spark.sql.types.{DataType, StructType} val newSchema = DataType.fromJson(jsonString).asInstanceOf[StructType]

the sample Schema I've been playing with has 300+feilds, for simplicity, I used the sample schema from here.

kode · Accepted Answer · 2019-08-10 16:31:26Z

1

SchemaConverter works for me. I used spark-shell to test and installed required package as spark-shell --packages "org.zalando:spark-json-schema_2.11:0.6.1".

scala> val schema = SchemaConverter.convertContent("""
 | {
 |   "$schema": "http://json-schema.org/draft-04/schema#",
 |   "title": "Product",
 |   "description": "A product from Acme's catalog",
 |   "type": "object",
 |   "properties": {
 |     "id": {
 |       "description": "The unique identifier for a product",
 |       "type": "integer"
 |     },
 |     "name": {
 |       "description": "Name of the product",
 |       "type": "string"
 |     },
 |     "price": {
 |       "type": "number",
 |       "minimum": 0,
 |       "exclusiveMinimum": true
 |     }
 |   },
 |   "required": [
 |     "id",
 |     "name",
 |     "price"
 |   ]
 | }
 | """)

schema: org.apache.spark.sql.types.StructType = StructType(StructField(id,LongType,false), StructField(name,StringType,false), StructField(price,DoubleType,false))

scala> schema.toString
res1: String = StructType(StructField(id,LongType,false), StructField(name,StringType,false), StructField(price,DoubleType,false))

Do you want to explicitly specify schema while reading json data?, because if you read json data using spark, it automatically infers schema from json data. eg.

val df = spark.read.json("json-file")
df.printSchema() // Gives schema of json data

answered Aug 10, 2019 at 16:31

kode

3841 silver badge2 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

btt3165 Over a year ago

The reason for wanting to specifically mention the schema is because I have one field that comes in sometimes as string when only 1 value is present and as an array when more than value is present. As a work around I was hoping that providing the schema may help read the field consistently. I tried this in Databricks, not sure if its Databricks setup is preventing the library from being called. Will try it in spark-shell and see if this works. Cheers.

user1119283 Over a year ago

Can someone point us an similar example in pyspark as well ? Thats is the way to convert such JSON schema to spark schema ?

Collectives™ on Stack Overflow

How to create DataFrame Schema from Json schem file

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related