Python running out of memory parsing XML using cElementTree.iterparse

Question

A simplified version of my XML parsing function is here:

import xml.etree.cElementTree as ET

def analyze(xml):
    it = ET.iterparse(file(xml))
    count = 0

    for (ev, el) in it:
        count += 1

    print('count: {0}'.format(count))

This causes Python to run out of memory, which doesn't make a whole lot of sense. The only thing I am actually storing is the count, an integer. Why is it doing this:

enter image description here

See that sudden drop in memory and CPU usage at the end? That's Python crashing spectacularly. At least it gives me a MemoryError (depending on what else I am doing in the loop, it gives me more random errors, like an IndexError) and a stack trace instead of a segfault. But why is it crashing?

stackoverflow.com/questions/1513592/… recommends calling .clear() on each element when you're done with it to save memory. Presumably this works because cElementTree keeps the previously-returned values in memory otherwise. — Wooble
– Wooble, Commented Oct 8, 2011 at 15:19
@Oliver lxml beats ElementTree, but not cElementTree when it comes to parsing. — Aillyn
– Aillyn, Commented Oct 8, 2011 at 20:25
@Wooble: In all 3 ElementTree implementations, iterparse() builds the tree. It is up to the caller to delete unwanted elements. — John Machin
– John Machin, Commented Oct 8, 2011 at 20:33
Just a note: this issue seems to not affect the memory on my Mac at all, but causes my Ubuntu server to hemorrhage RAM like it's going out of style. — Mike Davlantes
– Mike Davlantes, Commented Jun 12, 2020 at 18:52

jfs · Accepted Answer · 2012-11-07 01:00:41Z

6

Code example:

import xml.etree.cElementTree as etree

def getelements(filename_or_file, tag):
    context = iter(etree.iterparse(filename_or_file, events=('start', 'end')))
    _, root = next(context) # get root element
    for event, elem in context:
        if event == 'end' and elem.tag == tag:
            yield elem
            root.clear() # preserve memory

answered Nov 7, 2012 at 1:00

jfs

417k210 gold badges1k silver badges1.7k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Henrik Heimbuerger Over a year ago

Shouldn't you invoke clear() on elem as well? Or are you certain that just clearing the root will cause the garbage collector to collect the element as well?

jfs Over a year ago

@hheimbuerger: root.clear() is enough. I haven't dig to deep but the memory usage was small when I used it to parse large xml files.

Collectives™ on Stack Overflow

Python running out of memory parsing XML using cElementTree.iterparse

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related