C# ThreadPool - OutOfMemoryException

Question

I am messing around with ThreadPools for the first time for a project and need to be able to queue work items in two different sections of the main method. I made this simple example to test and I keep getting an OutOfMemoryException inside either for loop on the object creation line. Is there a sort of cleanup I need to perform at the end of the execution of a thread in order to free up any memory taken by that thread? If not how do I solve my memory issue?

class Program
{
    public static HashSet<String> domains;
    static void Main(string[] args)
    {
        //Static DB Connection
        //Database Set
        domains = new HashSet<string>();
        while (true)
        {
            //Pull Data From The Database
            String[] chars = { "a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k", "l" };
            for (int i = 0; i < 10; i++)
            {
                Loader l = new Loader(chars[i]);
                if (!domains.Contains(chars[i]))
                {
                    lock (domains)
                    {
                        domains.Add(chars[i]);
                    }
                    ThreadPool.QueueUserWorkItem(new WaitCallback(l.ThreadPoolCallback), i);
                }

            }
           for (int i = 0; i < 10; i++)
            {
                Parser p = new Parser();
                ThreadPool.QueueUserWorkItem(new WaitCallback(p.ThreadPoolCallback), i);
            }
        }
    }
}

Parser Class:

 class Parser
{
    public void ThreadPoolCallback(Object threadContext)
    {
        ManualResetEvent eventx = new ManualResetEvent(false);
        //int threadIndex = (int)threadContext;
        Console.WriteLine("Parser Thread: " + threadContext);
        eventx.Set();
    }
}

Loader Class:

class Loader
{
    String ch { set; get; }
    public Loader(String new_ch)
    {
        ch = new_ch;
    }
    public void ThreadPoolCallback(Object threadContext)
    {
        ManualResetEvent eventx = new ManualResetEvent(false);
        Console.WriteLine("Added " + this.ch + " to " + Thread.CurrentThread.GetHashCode());
        //int threadIndex = (int) threadContext;
        //Console.WriteLine("Loader Thread: " + threadContext + " " + ch + "  " + Thread.CurrentThread.GetHashCode());
        lock(Program.domains)
        {
            Program.domains.Remove(this.ch);
            Console.WriteLine("Removed " + this.ch + " from " + Thread.CurrentThread.GetHashCode());
        }
        eventx.Set();
    }

}

Thanks a bunch.

EDIT Thanks for all of the help. I now see the errors I was making

You have a permanent while loop that is creating threads until you run out of memory... No where do you exit that loop so it's just infinite. — test
– test, Commented Jun 27, 2012 at 22:30
I imagine it's the queue to the ThreadPool that's getting too large, not the # of threads it spawns... not that it really makes any difference. — spender
– spender, Commented Jun 27, 2012 at 22:34
your main loop does not defer, so in a single processor system, the threads never get chance to run, thus the queue keeps growing with nothing getting done, on a multiple process or system, it's likely the main loop is outpacing the threads executing the work items — Les
– Les, Commented Jun 27, 2012 at 22:46
Consider that the ThreadPool services a considerable number of other APIs... if you fill queues with work that can't be cleared immediately, other APIs will mysteriously start behaving strangely. Timers don't fire at the right time, async APIs only callback considerably later than completion etc. Beware thrashing the ThreadPool. — spender
– spender, Commented Jun 27, 2012 at 23:00

roken · Accepted Answer · 2012-06-27 22:35:36Z

2

You have an infinite while loop that does nothing but spit out new threads. Threads take a significant amount of memory (1MB in 32bit; 4MB in 64bit), so eventually you're going to blow out your memory if you don't exhaust your thread limit first.

Also -- your usage of the ManualResetEvent is incorrect. It needs to be created outside the scope of the method, otherwise it accomplishes nothing (because each thread that enters the method will create a new instance of the resetevent object; in order to synchronize, threads must be accessing the same object).

edited Jun 27, 2012 at 22:35

answered Jun 27, 2012 at 22:28

roken

4,0242 gold badges23 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

spender Over a year ago

Not exactly spitting out new threads... rather queueing a lot of work to the ThreadPool, which will ultimately (but not immediately) result in the # of ThreadPool threads creeping up.

roken Over a year ago

@spender True. So more specifically, the thread pool is going to grow until memory is exhausted, unless the thread limit is hit first.

spender Over a year ago

I think it more likely that the queue itself is where memory is being used rather than TP threads.

Sizzle Over a year ago

My understanding of the ThreadPool was that it creates lets say ten threads and then I enter my work into the queue and the number of threads stays constant as they work their way through the queue. So if I wanted to do an infinite loop like above pulling data from a database (which is being inserted by another process) and using that data to constantly put more work into the queue will I always run into this type of problem?

spender Over a year ago

Your assumption is incorrect. The ThreadPool will attempt to scale itself to the amount of work pending in its queue... in order to avoid creating lots of threads in response to transient peaks in queue length, the ThreadPool does not spin up new threads immediately in response to extra work, but demonstrates considerable latency when spinning out extra threads to deal with peak work loads.

|

Les · Accepted Answer · 2012-06-27 22:33:43Z

2

Though the threadpool limits the number of threads, your loop does not limit the number of user work items on the queue. Your infinite loop will exhaust mem.

answered Jun 27, 2012 at 22:33

Les

10.6k4 gold badges44 silver badges61 bronze badges

Comments

code4life · Accepted Answer · 2012-06-27 22:31:11Z

1

Like roken said, everything between the lines

while(true)
{
   ....
}

are getting repeated infinitely.

answered Jun 27, 2012 at 22:31

code4life

15.8k8 gold badges56 silver badges82 bronze badges

Collectives™ on Stack Overflow

C# ThreadPool - OutOfMemoryException

3 Answers 3

10 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

10 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related