Serialize big list of objects using newtonsoft json C#

Question

I have my object as below

public class CustomizedObject
{
    public int KeyId { get; set; }
    public string SomeName { get; set; }
    public DateTime StartDate { get; set; }
    public bool isPossible { get; set; }

    //Below dictionary is hug contains more than 300k rows (300*1000) 
    public Dictionary<string, SomBigObject> BigObjectMap { get; set; }

}

public class SomBigObject
{
    //This object has so many properties of all type (int, float, datetime, enum, double, bool) 
    //+ another customized user object Lists
}

I am trying to serialize this object using newtonsoft json library in C# using below settings.

var settings = new JsonSerializerSettings
            {
                PreserveReferencesHandling = PreserveReferencesHandling.Objects,
                NullValueHandling = NullValueHandling.Ignore,
                ReferenceLoopHandling = ReferenceLoopHandling.Serialize,
                Formatting = Formatting.Indented
            };
var json = JsonConvert.SerializeObject(value, settings);

But it gives me Out of memory exception. If i reduce number of rows to small number like 100k rows then it works fine.

When it has given out of memory exception, my computer has still 5 GB RAM available.

After serializing this object, i want to store it in Distributed Cache (Apache Ignite).

Please let me know what alternatives i can try to resolve this issue.

.net 4.5. is there improvement in any of recent version (>4.5.0)? — Gopu_Tunas
– Gopu_Tunas, Commented Jan 31, 2020 at 13:03

Alexandr Shapkin · Accepted Answer · 2020-01-31 10:54:41Z

1

Basically it's not quite clear what are you trying to achieve with the Ignite, but here are my observations. You don't need to serialize your data manually. Ignite is quite smart to perform those things for you. Moreover, it's counterproductive to save the data as a single batch of strings.

You could try to use the data colocation and have two caches - one for the CustomizedObject and the second with your SomBigObject [1] and fill the latter using a DataStreamer [2]. Regarding the serialization, Ignite stores all data using its own binary format [3] in order to provide required APIs, such as SQL for example. For a large custom object, it's a common practice to implement the IBInarizable interface [4] and get benefits out of the GetRawReader and GetRawWriter methods.

answered Jan 31, 2020 at 10:54

Alexandr Shapkin

2,4251 gold badge8 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Gopu_Tunas Over a year ago

There is bit performance issue with datetime fields when Ignite is serializing objects. apache-ignite-users.70518.x6.nabble.com/…

Alexandr Shapkin Over a year ago

Oh, I see. But there is a workaround with ForceTimestamp = true, isn't it? Furthermore, I haven't noticed any implementation of IBinarizable. The interface helps you to take over the serialization by yourself so that you can serialize your DateTime field in any suitable format, i.e. ticks, string, whatever.

Philip Atz · Accepted Answer · 2020-01-31 10:17:33Z

0

Your dictionary can be viewed as an IEnumerable<KeyValuePair<string, SomBigObject>>. Why don't you break that up into chunks of 100K rows and store them one chunk at a time?

answered Jan 31, 2020 at 10:17

Philip Atz

9481 gold badge10 silver badges26 bronze badges

Collectives™ on Stack Overflow

Serialize big list of objects using newtonsoft json C#

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related