Stack Overflow Exception in LINQ to SQL

Question

I have a method that based on some parameters finds 'Transactions' between two dates given a list of strings. When the list is > 1000, I get a stack overflow exception trying to iterate on the list.

Here is my code

public List<string> CodesWithTransactionsBetweenDates(DateTime startInclusive, DateTime endExclusive, List<string> associatedCodes, int marketId)
    {
        List<string> codesWithTransactionsInPeriod = new List<string>();
        using (var context = new MarketPlaceEntities())
        {
            var transactionList = (from transactions in context.Transactions
                                   where
                                    associatedCodes.Contains(transactions.User.Code) &&
                                    transactions.MarketId == marketId &&
                                    transactions.Date >= startInclusive &&
                                    transactions.Date < endExclusive
                                   group transactions by transactions.User.Code into uniqueIds
                                   select new { UserCode = uniqueIds.Key });
            foreach (var transaction in transactionList)
            {
                codesWithTransactionsInPeriod.Add(transaction.UserCode);
            }
            return codesWithTransactionsInPeriod;
        }
    }

Here is the stack trace... it goes past the point which visual studio can handle.

System.Data.Entity.dll!System.Data.Query.InternalTrees.BasicOpVisitor.VisitChildren(System.Data.Query.InternalTrees.Node n) + 0x3 bytes 
System.Data.Entity.dll!System.Data.Query.PlanCompiler.GroupAggregateRefComputingVisitor.VisitDefault(System.Data.Query.InternalTrees.Node n) + 0x2b bytes   
System.Data.Entity.dll!System.Data.Query.InternalTrees.BasicOpVisitor.VisitRelOpDefault(System.Data.Query.InternalTrees.RelOp op, System.Data.Query.InternalTrees.Node n) + 0xe bytes   
System.Data.Entity.dll!System.Data.Query.InternalTrees.BasicOpVisitor.VisitApplyOp(System.Data.Query.InternalTrees.ApplyBaseOp op, System.Data.Query.InternalTrees.Node n) + 0xe bytes  
System.Data.Entity.dll!System.Data.Query.InternalTrees.BasicOpVisitor.Visit(System.Data.Query.InternalTrees.OuterApplyOp op, System.Data.Query.InternalTrees.Node n) + 0xe bytes    
System.Data.Entity.dll!System.Data.Query.InternalTrees.OuterApplyOp.Accept(System.Data.Query.InternalTrees.BasicOpVisitor v, System.Data.Query.InternalTrees.Node n) + 0x10 bytes   
System.Data.Entity.dll!System.Data.Query.InternalTrees.BasicOpVisitor.VisitNode(System.Data.Query.InternalTrees.Node n) + 0x14 bytes    
System.Data.Entity.dll!System.Data.Query.InternalTrees.BasicOpVisitor.VisitChildren(System.Data.Query.InternalTrees.Node n) + 0x60 bytes

My question is what is a way I can handle this query so that I do not have to worry about a stack overflow exception?

What version of Linq are you using? There was a bug pre-4.0 that could cause this to happen: damieng.com/blog/2009/06/01/linq-to-sql-changes-in-net-40 "Contains now detects self-referencing IQueryable and doesn’t cause a stack overflow" — Matthew Watson
– Matthew Watson, Commented Feb 21, 2013 at 20:30
@MatthewWatson I am using .NET Framework 4.0 Thanks though :) — Kyle C
– Kyle C, Commented Feb 21, 2013 at 20:42
General suggestion for dealing with stack overflows: look at the call stack in a debugger to see what methods were involved. — Bryce Wagner
– Bryce Wagner, Commented Feb 21, 2013 at 21:20
The Stack overflow happens everytime in the ToList call... Is there some limitation to passing a list to ms sql ? — Kyle C
– Kyle C, Commented Feb 21, 2013 at 21:24

pwnyexpress · Accepted Answer · 2013-02-21 21:25:55Z

2

It seems like you're blowing the stack by iterating over the large collection, but also simultaneously add those objects to the list which results in two large, but basically identical collections. Instead just use AddRange for the list which accepts any IEnumerable.

List<string> codesWithTransactionsInPeriod = new List<string>();
using (var context = new MarketPlaceEntities())
    {
        return codesWithTransactionsInPeriod.AddRange((from transactions in context.Transactions
                               where
                                associatedCodes.Contains(transactions.User.Code) &&
                                transactions.MarketId == marketId &&
                                transactions.Date >= startInclusive &&
                                transactions.Date < endExclusive
                               group transactions by transactions.User.Code into uniqueIds
                               select uniqueIds.Key));
    }

or without instantiating an empty list...

using (var context = new MarketPlaceEntities())
    {
        return (from transactions in context.Transactions
                               where
                                associatedCodes.Contains(transactions.User.Code) &&
                                transactions.MarketId == marketId &&
                                transactions.Date >= startInclusive &&
                                transactions.Date < endExclusive
                               group transactions by transactions.User.Code into uniqueIds
                               select uniqueIds.Key).ToList<string>();
    }

or to preserve laziness... (Edited to use Lazy)

public Lazy<List<string>> LazyCodesWithTransactionsBetweenDates((DateTime startInclusive, DateTime endExclusive, List<string> associatedCodes, int marketId)
{
    return new Lazy<List<string>>(CodesWithTransactionsBetweenDates(startInclusive, endExclusive, associatedCodes, marketId));
}

private List<string> CodesWithTransactionsBetweenDates(DateTime startInclusive, DateTime endExclusive, List<string> associatedCodes, int marketId)
{
    using (var context = new MarketPlaceEntities())
    {
        return (from transactions in context.Transactions
                           where
                            associatedCodes.Contains(transactions.User.Code) &&
                            transactions.MarketId == marketId &&
                            transactions.Date >= startInclusive &&
                            transactions.Date < endExclusive
                           group transactions by transactions.User.Code into uniqueIds
                           select uniqueIds.Key).ToList<string>();
    }
}

edited Feb 21, 2013 at 21:25

answered Feb 21, 2013 at 20:28

pwnyexpress

1,0267 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

11 Comments

Kyle C Over a year ago

This doesn't compile, I get cannot convert type 'void' to 'List<string>'

pwnyexpress Over a year ago

See my edit for a better method that should compile. the first example may require the .AsEnumerable() extension method at the end of the inner linq query.

Kyle C Over a year ago

I did just try the second example and it does compile, but I still receive the StackOverflowException.

pwnyexpress Over a year ago

I think the problem you're facing here is really calling ToList(). Linq uses lazy evaluation and doesn't actually load objects into a collection until iterating over it, which is what ToList() does. I would suggest changing the method to return an IQueryable. See my 3rd edit...

Bryce Wagner Over a year ago

The "preserve laziness" last suggestion is inappropriate because context will be disposed before the expression is evaluated.

|

Sergey Berezovskiy · Accepted Answer · 2013-02-21 20:56:46Z

You have two big problems here - for each unique id key you are creating new object in memory with single property. Also you have useless local list, where you copying all those objects. Each time when list's capacity filled, new inner array is created and all objects copied there.

You can use streaming processing with IEnumerable. In this case you don't need to hold all data in memory:

public IEnumerable<string> CodesWithTransactionsBetweenDates(
         DateTime startInclusive, DateTime endExclusive, 
         List<string> associatedCodes, int marketId)
{
    // do not use local list

    using (var context = new MarketPlaceEntities())
    {
        return from transactions in context.Transactions
                where associatedCodes.Contains(transactions.User.Code) &&
                      transactions.MarketId == marketId &&
                      transactions.Date >= startInclusive &&
                      transactions.Date < endExclusive
                      group transactions by transactions.User.Code into uniqueIds
                      select uniqueIds.Key; // do not create anonymous object
    }
}

If you need list, you can apply ToList() on this query. But you definitely don't need create anonymous objects and copy them to local list.

Kyle C · Accepted Answer · 2013-02-21 22:41:16Z

0

Okay, after some trial and error and looking at some alternatives, I came up with a solution that seems to work out quite well.

public List<string> CodesWithTransactionsBetweenDates(DateTime startInclusive, DateTime endExclusive, List<string> associatedCodes, int marketId)
{
    using (var context = new MarketPlaceEntities())
    {
        var list = (from transactions in context.Transactions                            
                where
                    transactions.MarketId == marketId &&
                    transactions.Date >= startInclusive &&
                    transactions.Date < endExclusive                            
                select transactions.User.Code).Distinct().ToList<string>();

        return list.Where(c => associatedCodes.Contains(c)).ToList();
    }            
}

I imagine there is some sort of limitation with a list being used in the where clause, this ended up being a better solution as I'm limiting the User codes and then doing a simple filter to get only those that are in the associated codes list.

answered Feb 21, 2013 at 22:41

Kyle C

1,63712 silver badges25 bronze badges

1 Comment

pwnyexpress Over a year ago

Nice work on finding a work-around. I think you've highlighted a some of the things that the .NET team has to work on in regards to laziness in linq.

Collectives™ on Stack Overflow

Stack Overflow Exception in LINQ to SQL

3 Answers 3

11 Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

11 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related