Out of memory exception while using threads

Question

I have the following algorithm ,

private void writetodb()
{
    using(var reader = File.OpenRead("C:\Data.csv");
    using(var parser = new TextFieldParser(reader))
    { 
        //Do some opeartions
        while(!parser.EndOfData)
        {
            //Do operations
            //Take 500 rows of data and put it in dataset
            Thread thread = new thread(() => WriteTodb(tablename, set));
            thread.Start();
            Thread.Sleep(5000);
        }
    }
}

public void WriteTodb(string table, CellSet set)
{
    //WriteToDB
    //Edit: This statement will write to hbase db in hdinsight
    hbase.StoreCells(TableName, set);
}

This method works absolutely fine until 500 mb of data but after that it fails saying Out of memory exception.

I am pretty much sure that it is because of threads but using threads is mandatory and I cant change the architecture.
Can anybody tell me what modifications I have to make in thread programming in the above program to avoid memory exception.

A thread occupies approximately 1MB of memory. Given the amount of data (500MB!!!) your program probably runs out of memory. Consider using the Task Parallel Library instead of manually creating threads. — Cristian Lupascu
– Cristian Lupascu, Commented Jul 27, 2015 at 16:43
Are you sure that using threads is mandatory and that you can't use tasks instead? Also, are you sure the threads aren't contesting for the resource you're consuming (the file, I presume)? — Theodoros Chatzigiannakis
– Theodoros Chatzigiannakis, Commented Jul 27, 2015 at 16:44
What is the code of function "WriteTodb" ? How do manage your database connections? Perhaps because you use lots of database connections, — Matt
– Matt, Commented Jul 27, 2015 at 16:44
It is very inefficient to use threads for I/O; you should be using async/await. — Dour High Arch
– Dour High Arch, Commented Jul 27, 2015 at 17:07

VMAtm · Accepted Answer · 2017-04-07 19:55:38Z

First of all, I can't understand your words about threading:

I have to make in thread programming in the above program to avoid memory exception.

You will use the thread programming if you use the TPL, as it been already suggested. You really don't have to use the Thread class if you can't understand it. You say that your code is C# 4.0 so the TPL is an option for you. You can do you work something like this (very easy way):

List<Task> tasks  = new List<Task>();
while(!parser.EndOfData)
{
    tasks.Add(Task.Run(() => WriteTodb(tablename, set)));
}
Task.WaitAll(tasks.ToArray());

TPL engine will use the default TaskScheduler class, which uses internal ThreadPool and can level the resources you have on your server.

Also, I see that you're using the HBase client from Microsoft, and it has async method in it:

public async Task StoreCellsAsync(string table, CellSet cells)
{
}

So you can use the asynchronious approach in your code and TPL at the same time:

List<Task> tasks  = new List<Task>();
while(!parser.EndOfData)
{
    tasks.Add(WriteTodb(tablename, set)));
}
// asynchroniously await all the writes
await Task.WhenAll(tasks.ToArray());

public async Task WriteTodb(string table,CellSet set)
{
    //WriteToDB
    //Edit: This statement will write to hbase db in hdinsight asynchroniously!
    await hbase.StoreCellsAsync(TableName, set);
}

If, for some strange reasons, you can't use TPL, you have to refactor your code and write your own thread scheduler:

You don't have to create the thread for your write each time, you can reuse them.
Running second time inside the same thread is, in general, faster than create two different threads for each operation.
Split file into some parts, create thread for the writing, and write the data in a loop.

Deepak Bhatia · Accepted Answer · 2015-07-28 02:35:13Z

0

Instead of creating new Thread everytime use ThreadPool.QueueUserWorkItem. For refrence see this: https://msdn.microsoft.com/en-us/library/kbf0f1ct(v=vs.110).aspx

answered Jul 28, 2015 at 2:35

Deepak Bhatia

1,1201 gold badge8 silver badges13 bronze badges

1 Comment

Brian Rasmussen Over a year ago

That's good advice in general, but it doesn't really address the question.

Collectives™ on Stack Overflow

Out of memory exception while using threads

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related