Large data processing in x86 C# gives System.OutOfMemory exception

Question

I am processing XML coming from server which contains both images and data in one C# function (compiled 32 bit). When I try to parse this xml in memory it gives me System.OutOfMemory exception.

Is there any way to avoid this error?

My guess is, system cannot find contiguous block of 50-100MB memory. (my pc hv 8Gig ram and its quad core)

Are you saying the XML files are 50-100mb each? What XML parser are you using? Post your code — Michael Haren
– Michael Haren, Commented Apr 16, 2010 at 2:41
You say this is a server app - how many concurrent xml docs are you parsing ? Maybe you can limit how many inbound requests get processed concurrently on the server ? — Romain Hippeau
– Romain Hippeau, Commented Apr 16, 2010 at 3:21
@Michael, I am getting one XElement at a time which I get using following code- Encoding.UTF8.GetString(Convert.FromBase64String(xelement.Value.ToString()); Then from this output, I process each node individually. @Romain, I process only one xml doc at a time. — Dipak
– Dipak, Commented Apr 16, 2010 at 21:32

Josh · Accepted Answer · 2010-04-16 03:07:48Z

2

An XML file that is 50-100 MB on disk is going to be a lot larger when parsed into a DOM. (Assuming of course that you're using XmlDocument or XDocument.)

Although I despise the API, you may want to look into using XmlReader to "stream" the document in. It will be much more performant than using the DOM but it's a lot less intuitive to use.

answered Apr 16, 2010 at 3:07

Josh

69.4k15 gold badges149 silver badges158 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Large data processing in x86 C# gives System.OutOfMemory exception

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related