json reference extraction in python

Question

Json is not only useful as a communication tool for APIs, but also may be used as a markup for configuring running programs as initialization.

I encountered the use of references in json schema for the purpose of reuse.

Since json schema is valid json, I had expected the python json library to have the ability to expand references.

$ cat test.json
{ 
  "template":{
    "a":"a",
    "b":"b",
    "pi":3.14
  },
  "value": { "$ref":"#/template"}
}
python -c "from json import load; fp = open(\"test.json\",\"r\"); print(load(fp))"
{'template': {'a': 'a', 'b': 'b', 'pi': 3.14}, 'value': {'$ref': '#/template'}}

What is the simplest way to expand the references in python, since python dicts cannot point to other parts of themselves (I think)?

What makes you think "python dicts cannot point to other parts of themselves"? Actually python dict values are just pointers, so you can have multiple keys pointing to the same value, and also cyclical references. This isn't really supported in standard json, though. I guess you are using some json superset? — Håken Lid
– Håken Lid, Commented Oct 18, 2018 at 19:25
Json references are indeed not in the standard, but an expired Internet draft. However, they are used in json-schema and swagger, which is where I came across them. @anton I don't know man. Maybe it was an unrealistic expectation to make. But I had a usecase for it - and it is nice when you find someone else has already solved the problem. — Debanjan Basu
– Debanjan Basu, Commented Oct 19, 2018 at 5:37

Hugues Fontenelle · Accepted Answer · 2018-10-18 11:20:48Z

10

The json library does not support references, but jsonref does.

jsonref is a library for automatic dereferencing of JSON Reference objects for Python (supporting Python 2.6+ and Python 3.3+).

From the docs:

from pprint import pprint
import jsonref

# An example json document
json_str = """{"real": [1, 2, 3, 4], "ref": {"$ref": "#/real"}}"""
data = jsonref.loads(json_str)
pprint(data)  # Reference is not evaluated until here
{'real': [1, 2, 3, 4], 'ref': [1, 2, 3, 4]}

answered Oct 18, 2018 at 11:20

Hugues Fontenelle

5,4453 gold badges31 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Debanjan Basu Over a year ago

I have to give it to you.. this is the perfect answer! :)

ResponsiblyUnranked Over a year ago

Note to people using this library, it doesn't seem to be able to handle escaped characters in references like ~1 for / characters. I've raised an issue on their GitHub.

Debanjan Basu · Accepted Answer · 2018-10-18 10:53:58Z

1

thanks to @FreshD , who led me towards YAML which can be loaded and dumped just like JSON.

This is how I solved my use case, which was exact substitution to start with, but now also supports inheritance.

$ cat test.yaml 
template: &item
    a: a
    b: b
    pi: 3.14
exact-value: 
    *item
inherited-value:
    <<: *item
    a : I changed
$ python -c "from yaml import load; fp = open(\"test.yaml\",\"r\"); print(load(fp))"
{'template': {'a': 'a', 'b': 'b', 'pi': 3.14}, 'exact-value': {'a': 'a', 'b': 'b', 'pi': 3.14}, 'inherited-value': {'a': 'I changed', 'b': 'b', 'pi': 3.14}}

answered Oct 18, 2018 at 10:53

Debanjan Basu

9141 gold badge8 silver badges31 bronze badges

1 Comment

Anthon Over a year ago

PyYAML's load() is documented to be potentially unsafe. If you use PyYAML use safe_load(). You should consider using my ruamel.yaml, it implements YAML 1.2 (PyYAML only does 1.1) and that is a superset of JSON, so it can load both. python -c "from ruamel.yaml import YAML; print(YAML(typ='safe').load(open('test.yaml'))). There is no need for escaped double quotes. Use single quotes within the command string.

FreshD · Accepted Answer · 2018-10-18 10:00:50Z

0

I never used/saw these references in json, but I know and used the in YAML also for configuration. This is a nice description how to achieve this.

edited Oct 18, 2018 at 10:00

answered Oct 18, 2018 at 9:34

FreshD

3,1522 gold badges28 silver badges36 bronze badges

2 Comments

Debanjan Basu Over a year ago

thanks for responding.. but your answer showed how to use variables. I didn't like the mixing of code/directives inside the data file. but you did point me in the correct direction. blog.daemonl.com/2016/02/yaml.html

FreshD Over a year ago

This page was the one I originally wanted to show you, but was not able to find it anymore. I also used this as reference.

Collectives™ on Stack Overflow

json reference extraction in python

3 Answers 3

2 Comments

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related