Improve performance when sorting and reading JSON from file C#

Question

I have a JSON file that contains about 20k lines of code that has to be read, sorted and saved into a database. I've written code for it and it works the way it's suppose to but my issue is that it takes about 10 minutes. Therefor I wonder if someone has any ideas what can be done to enhance the performance?

Json:

{
    "Number": 123456,
    "Area": "NE01"
},
{
    "Number": 123457,
    "Area": "NE01"
},

and so forth....

C#:

dynamic json = JsonConvert.DeserializeObject(File.ReadAllText(path, Encoding.UTF8));

foreach (var obj in json)
{
    if (obj.Area == "NE01")
    {
        var o = new object
        {
            Number = obj.Number,
        };
        db.Entity.Add(obj);
        continue;
    }

    if (obj.Area == "NE02")
    {
        var o = new object
        {
            Number = obj.Number,
        };
        db.Entity.Add(obj);
        continue;
    }

    if (obj.Area == "NE03")
    {
        var o= new object
        {
            Number = obj.Number,
        };
        db.Entity.Add(obj);
        continue;
    }

    if ( obj.Area== "NE04")
    {
        var o = new object
        {
            Number = obj.Number
        };
        db.Entity.Add(obj);
        continue;
    }
}

db.SaveChanges();

To make it clearer, area has four different values. Depending on the value the number will have a foreign key that points to the area. Unfortunately I'm not allowed to change anything in the underlying database. Let me know if I have to provide further information.

Have you added timings to see where the time is going? there are 4 places to time, as a start: File.ReadAllText, JsonConvert.DeserializeObject, the foreach loop, and finally SaveChanges. Until you know where the time is being spent: you can't start to make it faster. — Marc Gravell
– Marc Gravell, Commented Feb 14, 2019 at 10:05
might take a look at the bulk operations - EF do not support it. In case SaveChanges is slow — Vladimir
– Vladimir, Commented Feb 14, 2019 at 10:06
@SaniSinghHuttunen I think we can infer that it is an extract from the JSON, presumably from a JSON array; I'm not sure that's something we need to get hung up on... — Marc Gravell
– Marc Gravell, Commented Feb 14, 2019 at 10:09
to echo @Vladimir: if all you're doing here is insert, then SqlBulkCopy may be your best option. There's interesting ways of doing that, but perhaps the most convenient (if you have a List<T> of a T that looks like the table) is to use FastMember.ObjectReader look for bcp here: github.com/mgravell/fast-member — Marc Gravell
– Marc Gravell, Commented Feb 14, 2019 at 10:10

Sani Huttunen · Accepted Answer · 2019-02-14 10:28:26Z

1

Using EntityFramework.Utilities you can use Bulk Insert which should speed up insertions.

Something like:

public class Data
{
  public int Number { get; set; }
  public string Area { get; set; }
}

var objects = JsonConvert.DeserializeObject<List<Data>>(File.ReadAllText(path, Encoding.UTF8))
  .Select(d => new object { Number = d.Number })
  .ToList();

EFBatchOperation.For(db, db.Entity).InsertAll(objects);

Disclaimer: Code not tested.

answered Feb 14, 2019 at 10:28

Sani Huttunen

24.4k7 gold badges78 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

LordSilvermort · Accepted Answer · 2019-02-20 10:53:00Z

0

Only by using AddRange I was able to decrease the time to just over one minute. Which is good enough for my purpose.

answered Feb 20, 2019 at 10:53

LordSilvermort

2772 silver badges11 bronze badges

Collectives™ on Stack Overflow

Improve performance when sorting and reading JSON from file C#

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related