Serializing heavily linked data in .NET (customizing JSON.NET references)

Question

I want to avoid reinventing the wheel when serializing data. I know some ways to serialize objects which are linked to each other, but it ranges from writing some code to writing a lot of code for serialization, and I'd like to avoid that. There must be some generic solutions.

Let's say I have a structure like this:

Person
    bro = new Person { name = "bro", pos = new Pos { x = 1, y = 5 } },
    sis = new Person { name = "sis", pos = new Pos { x = 2, y = 6 } },
    mom = new Person { name = "mom", pos = new Pos { x = 3, y = 7 }, 
        children = new List<Person> { bro, sis }
    },
    dad = new Person { name = "dad", pos = new Pos { x = 4, y = 8 }, 
        children = new List<Person> { bro, sis }, mate = mom
    };
mom.mate = dad;
Family family = new Family { persons = new List<Person> { mom, dad, bro, sis } };

I want to serialize data to something like this:

family: {
    persons: [
        { name: "bro", pos: { x: 1, y: 5 } },
        { name: "sis", pos: { x: 2, y: 6 } },
        { name: "mom", pos: { x: 3, y: 7 }, mate: "dad", children: [ "bro", "sis" ] },
        { name: "dad", pos: { x: 4, y: 8 }, mate: "mom", children: [ "bro", "sis" ] },
    ]
}

Here, links are serialized as just names, with the assumption that names are unique. Links can also be "family.persons.0" or generated unique IDs or whatever.

Requirements:

Format must be human-readable and preferably human-writable too. So, in order of preference: JSON, YAML*, XML, custom. No binary formats.
Serialization must support all good stuff .NET offers. Generics are a must, including types like IEnumerable<>, IDictionary<> etc. Dynamic types / untyped objects are desirable.
Format must not be executable. No Lua, Python etc. scripts and things like that.
If unique IDs are generated, they must be stable (persist through serialization-deserialization), as files will be put into a version control system.

* Heard about YAML, but sadly, it seems to be pretty much dead.

Did you look at DataContractSerializer, JavaScriptSerializer and JSON.NET? — Oded
– Oded, Commented Oct 28, 2012 at 12:13
@Oded DataContractSerializer fails with stack overflow, JavaScriptSerializer throws an exception after detecting a circular reference... JSON.NET's website says, "Serializes circular references", so I'm going to try it. — Athari
– Athari, Commented Oct 28, 2012 at 12:42
Why do you want your serialized data to be in a very specific structure? Why not use the existing serialization framework and let them do their thing? (Since you want readable, use XML (I think SOAP is also "human readable")) — LightStriker
– LightStriker, Commented Oct 28, 2012 at 13:19
Getting errors from existing serialization support is a very poor reason to invent your own. After doing a lot of work, you'll just fall into the exact same trap again. Post real code to get help. — Hans Passant
– Hans Passant, Commented Oct 28, 2012 at 13:37
@Oded Okay, JSON.NET does support circular references, but it always expands objects when it first encounters them and there's no way to change this behavior (at least I don't see how to). This is what I got using custom IReferenceResolver: pastebin.com/tPVBELen — Athari
– Athari, Commented Oct 28, 2012 at 14:27

Athari · Accepted Answer · 2012-10-28 22:59:40Z

12

Solved the problem using JSON.NET (fantastic library!). Now objects are, first, serialized and referenced exactly where I want them them to; and second, without numerous "$id" and "$ref" fields. In my solution, the first property of an object is used as its identifier.

I've created two JsonConvertors (for references to objects and for referenced objects):

interface IJsonLinkable
{
    string Id { get; }
}

class JsonRefConverter : JsonConverter
{
    public override void WriteJson (JsonWriter writer, object value, JsonSerializer serializer)
    {
        writer.WriteValue(((IJsonLinkable)value).Id);
    }

    public override object ReadJson (JsonReader reader, Type type, object existingValue, JsonSerializer serializer)
    {
        if (reader.TokenType != JsonToken.String)
            throw new Exception("Ref value must be a string.");
        return JsonLinkedContext.GetLinkedValue(serializer, type, reader.Value.ToString());
    }

    public override bool CanConvert (Type type)
    {
        return type.IsAssignableFrom(typeof(IJsonLinkable));
    }
}

class JsonRefedConverter : JsonConverter
{
    public override void WriteJson (JsonWriter writer, object value, JsonSerializer serializer)
    {
        serializer.Serialize(writer, value);
    }

    public override object ReadJson (JsonReader reader, Type type, object existingValue, JsonSerializer serializer)
    {
        var jo = JObject.Load(reader);
        var value = JsonLinkedContext.GetLinkedValue(serializer, type, (string)jo.PropertyValues().First());
        serializer.Populate(jo.CreateReader(), value);
        return value;
    }

    public override bool CanConvert (Type type)
    {
        return type.IsAssignableFrom(typeof(IJsonLinkable));
    }
}

and a context to hold references data (with a dictionary for each type, so IDs need to be unique only among objects of the same type):

class JsonLinkedContext
{
    private readonly IDictionary<Type, IDictionary<string, object>> links = new Dictionary<Type, IDictionary<string, object>>();

    public static object GetLinkedValue (JsonSerializer serializer, Type type, string reference)
    {
        var context = (JsonLinkedContext)serializer.Context.Context;
        IDictionary<string, object> links;
        if (!context.links.TryGetValue(type, out links))
            context.links[type] = links = new Dictionary<string, object>();
        object value;
        if (!links.TryGetValue(reference, out value))
            links[reference] = value = FormatterServices.GetUninitializedObject(type);
        return value;
    }
}

Some attributes on the properties are necessary:

[JsonObject(MemberSerialization.OptIn)]
class Family
{
    [JsonProperty(ItemConverterType = typeof(JsonRefedConverter))]
    public List<Person> persons;
}

[JsonObject(MemberSerialization.OptIn)]
class Person : IJsonLinkable
{
    [JsonProperty]
    public string name;
    [JsonProperty]
    public Pos pos;
    [JsonProperty, JsonConverter(typeof(JsonRefConverter))]
    public Person mate;
    [JsonProperty(ItemConverterType = typeof(JsonRefConverter))]
    public List<Person> children;

    string IJsonLinkable.Id { get { return name; } }
}

[JsonObject(MemberSerialization.OptIn)]
class Pos
{
    [JsonProperty]
    public int x;
    [JsonProperty]
    public int y;
}

So, when I serialize and deserialize using this code:

JsonConvert.SerializeObject(family, Formatting.Indented, new JsonSerializerSettings {
    NullValueHandling = NullValueHandling.Ignore,
    Context = new StreamingContext(StreamingContextStates.All, new JsonLinkedContext()),
});

JsonConvert.DeserializeObject<Family>(File.ReadAllText(@"..\..\Data\Family.json"), new JsonSerializerSettings {
    Context = new StreamingContext(StreamingContextStates.All, new JsonLinkedContext()),
});

I get this neat JSON:

{
  "persons": [
    {
      "name": "mom",
      "pos": {
        "x": 3,
        "y": 7
      },
      "mate": "dad",
      "children": [
        "bro",
        "sis"
      ]
    },
    {
      "name": "dad",
      "pos": {
        "x": 4,
        "y": 8
      },
      "mate": "mom",
      "children": [
        "bro",
        "sis"
      ]
    },
    {
      "name": "bro",
      "pos": {
        "x": 1,
        "y": 5
      }
    },
    {
      "name": "sis",
      "pos": {
        "x": 2,
        "y": 6
      }
    }
  ]
}

What I don't like in my solution, is that I have to use JObject, even though technically it's unnecessary. It probably creates quite a bit of objects, so loading will be slower. But looks like this is the most widely used approach for customizing convertors of objects. Methods which could be used to avoid this are private anyway.

answered Oct 28, 2012 at 22:59

Athari

33.9k17 gold badges111 silver badges153 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

misnomer Over a year ago

I'm confused how the JsonRefed converter works - every time I try to pass onwards to "serializer.Serialize(writer, value);" I get a circular reference error as it tries to then call the same converter again for the same object?

Athari Over a year ago

@misnomer RefedConverter writes and reads as if no converter was used, but remembers ids of objects it reads. RefConverter writes just id strings and converts strings to objects using info stored by RefedConverter. You can try disabling circular references check in serializer settings. If you use both converters correctly (one primary source is "refed", the rest are "ref"), it should not crash.

pek Over a year ago

Is there a way for you to do this without having to set attributes in each field? Like, is there a way to set things in the ConverterSettings and then let everything work kind of automatically?

Athari Over a year ago

@pek You can move configuration anywhere you want to modifying contract resolver, context and converters, but the configuration has to exist somewhere, as only only the programmer knows where links and values are. If you want serialization to work "automatically", you can use built-in referece functionality, but you'll obviously lose the ability to choose where links and values are stored.

pek Over a year ago

@Athari yeah, well, the problem I'm having is that let's say I want to deserialize a list of Person and I set the refconverter in the settings (there is no property that uses it). The problem is that the converter will be used both when deserializing an entire person AND when a class member has a person reference. I only want the converter to be used in the second case, not the first. I hope you understand what I mean.

|

Collectives™ on Stack Overflow

Serializing heavily linked data in .NET (customizing JSON.NET references)

1 Answer 1

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related