1

I'm seeing my memory usage getting too hign on Diagnostic Tools after I call some endpoints.

I tried to isolate the problem in the smallest chunk I could, so I could eliminate all other factors, and I ended up with the following:

    [HttpGet("test")]
    public ActionResult Test()
    {
        var results = _context.Products
            .Include(x => x.Images)
            .Include(x => x.Options)
                .ThenInclude(x => x.Lists)
                    .ThenInclude(x => x.PriceChangeRule)
            .Include(x => x.Options)
                .ThenInclude(x => x.Lists)
                    .ThenInclude(x => x.Items)
                        .ThenInclude(x => x.PriceChangeRule)
            .Include(x => x.Options)
                .ThenInclude(x => x.Lists)
                    .ThenInclude(x => x.Items)
                        .ThenInclude(x => x.SupplierFinishingItem)
                            .ThenInclude(x => x.Parent)
            .Include(x => x.Category)
                .ThenInclude(x => x.PriceFormation)
                    .ThenInclude(x => x.Rules)
            .Include(x => x.Supplier)
                .ThenInclude(x => x.PriceFormation)
                    .ThenInclude(x => x.Rules)
            .Include(x => x.PriceFormation)
                .ThenInclude(x => x.Rules)
                .AsNoTracking().ToList();

        return Ok(_mapper.Map<List<AbstractProductListItemDto>>(results));
    }

It is a big query, with lots of includes, but the amount of data returned from the database is not huge, it's ~10.000 items. When I serialize this result it has only 3.5Mb.

My API is using around 300Mb of memory, then when I call this test endpoint, this value goes to about 1.2Gb. I think this is too much for only 3.5Mb of data, but I don't know how EF Core works internally, so I'll just ignore it.

My problem is that, as far as I understand, the DbContext is added as a scoped service, so it's created when the request starts and then killed when it finishes. Here's how I'm registering it:

    services.AddDbContext<DatabaseContext>(options =>
            options.UseSqlServer(
                Configuration.GetConnectionString("DefaultConnection")));

If my understanding is correct, when the request is finished, that huge amount of memory should be disposed, right?

The problem is that my memory usage never goes back again, I tried disposing the context manually, calling the garbage collector manually too, but the memory stays at 1.2Gb.

Am I missing something here?

7
  • 2
    The database context might be removed but not the whole EF backend. Most likely, it is caching the results. Commented May 11, 2019 at 18:02
  • 1
    There are a lot of potential ways that memory could be getting allocated here... Some things I can think of could be (1) you appear to be using Automapper, there could be some weird logic in the mapping making more allocations; (2) your EF objects may have far more data attached than the final mapped/serialized object (you may not be serializing all fields); or (3) you may have 10,000 products but their "images", "options", "rules", or etc could be far more numerous. It looks like you'll most likely have to profile the code to really get at the cause. Commented May 11, 2019 at 18:14
  • @cmcquillan The original objetcs have more data than the serialized result, but the entire database has 200Mb so I still think that 1.2Gb is a bit too much. I really didnt thought on Automapper. I'll try to remove it from the equation too. I tried to take a memory snapshot from Diagnostic Tools, but it didn't tell me anything useful :( Commented May 11, 2019 at 18:27
  • @gerwin Could it be caching anything even when I'm using .AsNoTracking()? Commented May 11, 2019 at 18:38
  • Honestly I am not entirely sure. It might be that some of EF caching is done on a lower level but you'd have to do some serious performance profiling to figure that out. I'd look at your problem from a different angle: for what reason are you loading 10.000 records with many joins and returning them in a Web API call at all? Honestly it's hard to come up with a valid use case for such a thing... Commented May 11, 2019 at 18:43

1 Answer 1

4

A potential area I can see is that you may be loading far more data out of the database than is required by the serialization to AbstractProductListItemDto. For example, your Product might have fields like below:

public class Product
{
    public string Id { get; set; }
    public string Name { get; set; }
    public string Description { get; set; }
}

However, your final DTO may only have one or two of those properties, for example:

public class AbstractProductListItemDto
{
    public string ProductId { get; set; }
    public string ProductName { get; set; }
}

This may also be true for the other tables which you are including (Options, Lists, Rules, etc), especially tables which are one-to-many which could easily explode the number rows/columns being queried.

A potential way to optimize this is to do the projection yourself as part of the LINQ query. This would take advantage of a feature of EF Core where it only selects the columns from the database which you have specified. For example:

This would select all columns from the product table

var results = _context.Products.ToList();

This would select only the Id and Name columns from the product table, resulting in less memory usage

var results = _context.Products.Select(x => new ProductDto { 
    Id = x.Id,
    Name = x.Name,
}

From the question I do not know all of the properties on all of the items which you are mapping, so it would be up to you if you wanted to do that mapping manually. The critical part is that you would need to do that in a call to Select() before your call to ToList() on your query.

However, there is a potential shortcut if you are using Automapper

Automapper includes a shortcut which attempts to write these query projections for you. It may not work depending on how much additional logic is happening within Automapper, but it might be worth a try. You would want to read up on the ProjectTo<>() method. If you were using projection, the code would probably look something like this:

Edit: It was correctly pointed out in comments that the Include() calls are not needed when using ProjectTo<>(). Here is a shorter sample with the original included below it

Updated:

using AutoMapper.QueryableExtensions;
// ^^^ Added to your usings
// 

    [HttpGet("test")]
    public ActionResult Test()
    {
        var projection = _context.Products.ProjectTo<AbstractProductListItemDto>(_mapper.ConfigurationProvider);

        return Ok(projection.ToList());
    }

Original:

using AutoMapper.QueryableExtensions;
// ^^^ Added to your usings
// 

    [HttpGet("test")]
    public ActionResult Test()
    {
        var results = _context.Products
            .Include(x => x.Images)
            .Include(x => x.Options)
                .ThenInclude(x => x.Lists)
                    .ThenInclude(x => x.PriceChangeRule)
            .Include(x => x.Options)
                .ThenInclude(x => x.Lists)
                    .ThenInclude(x => x.Items)
                        .ThenInclude(x => x.PriceChangeRule)
            .Include(x => x.Options)
                .ThenInclude(x => x.Lists)
                    .ThenInclude(x => x.Items)
                        .ThenInclude(x => x.SupplierFinishingItem)
                            .ThenInclude(x => x.Parent)
            .Include(x => x.Category)
                .ThenInclude(x => x.PriceFormation)
                    .ThenInclude(x => x.Rules)
            .Include(x => x.Supplier)
                .ThenInclude(x => x.PriceFormation)
                    .ThenInclude(x => x.Rules)
            .Include(x => x.PriceFormation)
                .ThenInclude(x => x.Rules)
                .AsNoTracking(); // Removed call to ToList() to keep it as IQueryable<>

        var projection = results.ProjectTo<AbstractProductListItemDto>(_mapper.ConfigurationProvider);

        return Ok(projection.ToList());
    }
Sign up to request clarification or add additional context in comments.

2 Comments

Includes are not needed. ProjectTo will fetch by default everything needed in the DTO.
@LucianBargaoanu Good catch. I've updated the answer.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.