how to write validator method to validate the json element data with python

Question

I'm new to python and trying to writing a python script using jsonschema to validate a huge json output file's schema. Want to make sure my json file doesn't have any null values in it.

Wrote a method to read the schema json file and the output json file, now passed them both to validate function. There are many repeating objects in the json file. Then realized that I should write validator function/class to pass each object and keep validating them in a loop.But stuck here not sure how to do that

{
  "id": "test",
  "name": "name",
  "cake_name": "test",
  "metric": 0.5,
  "anticipations": [
    {
      "time": "2018-01-01 00:00:00",
      "points": 0.49128797804879504,
      "top_properties": {
        "LA:TB2341": 0.23,
        "LA:TB2342": 0.23,
        "LA:TB2343": 0.23
      },
      "status": 0,
      "alert": false
    },
    {
      "time": "2018-01-02 00:00:00",
      "points": 0.588751186433263,
      "top_properties": {
        "LA:TB2342": 0.23,
        "LA:TB2341": 0.23,
        "LA:TB2344": 0.23
      },
      "status": 0,
      "alert": true
    }
  ]
}

PS: corresponding schema file is generated from "https://jsonschema.net/" which is my moduleschema.json and above json is modelout.json

Code I wrote just to read files:

def test_json(self):
    with open('/Users/moduleschema.json', 'r') as json_file:

        schema = json_file.read()
        print(schema)

    with open('/Users/modelout.json', 'r') as output_json:
        outputfile = output_json.read()

        print(outputfile)

    strt = jsonschema.validate(outputfile, schema)
    jsonschema.Draft4Validator
    print(strt)

I want to parse through the json file to make sure all the fields display the right types(ints for ints, strings for string values). I'm a newbie in python forgive me if this is a silly question. thanks!

Buckeye14Guy · Accepted Answer · 2019-09-11 21:34:54Z

So I am going to give an answer that relies on a 3rd party package that I really really like. I did not contribute to it but I have used it and it is very useful especially for the type of validation here.

Yes you can create a custom validator like

import json
import typing

# here json_data is the data in your question
def custom_validator(json_data: typing.Dict):
    string_attributes = ["id", "name", "cake_name", "status", "time", "LA:TB2342", "LA:TB2341", "LA:TB2344"]
    int_attributes = [...]
    float_attributes = [...]
    validations_errors = []
    for attribute in string_attributes:
        if attribute in json_data:
            if attribute in string_attributes and not isinstance(json_data.get(attribute), str):
                validations_errors.append(f"key {attribute} is not a string, got {json_data.get(attribute)}")
                ...

This can quickly get out of hand. Perhaps you can spend more time to make it pretty etc.

BUT, I highly suggest that you read up on dataclasses and pydantic

Here is the solution I would use

import json
import typing
from pydantic import BaseModel

# if you look closely, this just represents those tiny dictionaries in your list
class Anticipation(BaseModel):
    time: str
    points: float
    top_properties: typing.Dict[str, float]
    status: int
    alert: bool

# this is the whole thing, note how we say that anticipations is a list of those objects we defined above
class Data(BaseModel):
    id: str
    name: str
    cake_name: "str"
    metric: float
    anticipations: typing.List[Anticipation]

json_data = """{
  "id": null,
  "name": "name",
  "cake_name": "test",
  "metric": 0.5,
  "anticipations": [
    {
      "time": "2018-01-01 00:00:00",
      "points": 0.49128797804879504,
      "top_properties": {
        "LA:TB2341": 0.23,
        "LA:TB2342": 0.23,
        "LA:TB2343": 0.23
      },
      "status": 0,
      "alert": false
    },
    {
      "time": "2018-01-02 00:00:00",
      "points": 0.588751186433263,
      "top_properties": {
        "LA:TB2342": 0.23,
        "LA:TB2341": 0.23,
        "LA:TB2344": 0.23
      },
      "status": null,
      "alert": true
    }
  ]
}
"""
data = json.loads(json_data)
data = Data(**data)

I changed id to null and status to null in the last anticipation. If you run this, it will fail and show you this message. Which is fairly useful

pydantic.error_wrappers.ValidationError: 2 validation errors
id
  none is not an allowed value (type=type_error.none.not_allowed)
anticipations -> 1 -> status
  value is not a valid integer (type=type_error.integer)

Obviously this means that you will have to install a 3rd party package and for a new python coder people would suggest not to do that. In that case, the template below should point you in the right direction

def validate(my_dict: typing.Dict, string_attributes, int_attributes, float_attributes):
    validations_errors = []
    for attribute in string_attributes:
        if attribute in my_dict:
            if attribute in string_attributes and not isinstance(my_dict.get(attribute), str):
                validations_errors.append(f"key {attribute} is not a string, got {my_dict.get(attribute)}")
            if attribute in int_attributes and not isinstance(my_dict.get(attribute), int):
                # append to the list of errors
                pass
    return validations_errors


def custom_validator(json_data: typing.Dict):
    string_attributes = ["id", "name", "cake_name", "time", "LA:TB2342", "LA:TB2341", "LA:TB2344"]
    int_attributes = [...]
    float_attributes = [...]

    # now do it for anticipations
    validation_errors = validate(json_data, string_attributes, int_attributes, float_attributes)
    for i, anticipation in enumerate(json_data.get('anticipations')):
        validation_error = validate(anticipation, string_attributes, int_attributes, float_attributes)
        if validation_error:
            validation_errors.append(f"anticipation -> {i} error: {validation_error}")

    return validation_errors

data = json.loads(json_data)
custom_validator(data)

Output: ['key id is not a string, got None']

You can build up on that function

Collectives™ on Stack Overflow

how to write validator method to validate the json element data with python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related