Efficient way to perform xml parsing in python

Question

I have an XML file of(30GB) which contains 2 classes of data, The data of class 1 has corresponding

<id="11" class="1" bestmatchingid="50" Body="abc"> </id>
.
.
.
<id="9999890" class="2" MatchingClass1Id="11" Body="xyz"></id>

Now the task is to extract class1's body and corresponding class 2's body where e.g.

class1's id(11)== MatchingClass1Id of class2(which is 9999890)

I am accomplishing the same by using string comparison's in Python...is there a more efficient way in Python to accomplish the same considering my file size is 30 GB

Why have you tagged regex? XML Parsers are the best way to go... — jamylak
– jamylak, Commented Apr 5, 2012 at 9:53
I am attempting using regex, therefore tagged regex...but any efficient way would work for me. — Jannat Arora
– Jannat Arora, Commented Apr 5, 2012 at 9:54

Fred Foo · Accepted Answer · 2012-04-05 09:54:15Z

4

Use LXML's iterparse function. See the IBM DeveloperWorks article about it for how to use it on very large files.

answered Apr 5, 2012 at 9:54

Fred Foo

365k80 gold badges765 silver badges852 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Jannat Arora Over a year ago

Thanks a ton for helping. Can you please direct me to a tutorial which may help a beginner like me to catch up with the basics of lxml. Also in the IBMDeveloperWorks article...does not work for my code...since i have "id, class, Body, matchingclass1Id " and they guys are working with title...may be since i am novice i did not understand the subtilities...a big apology for disturbing again...but still despite ur helping i m stuck with trying to understand the basics

Fred Foo Over a year ago

@user869790: what exactly is the problem, lack of Python knowledge or lack of XML knowledge?

Jannat Arora · Accepted Answer · 2012-04-05 19:50:23Z

-1

lxml works good for your purpose. Also since you are a begineer..so for understanding the basic refer to the tutorial:

http://infohost.nmt.edu/tcc/help/pubs/pylxml/web/etree-view.html

All iterparse method is an efficient method to solve your problem

answered Apr 5, 2012 at 19:50

Jannat Arora

3,0098 gold badges47 silver badges74 bronze badges

1 Comment

Lennart Regebro Over a year ago

(Completely OT: Hah. I was just to answer the CSV question when you deleted it. I'm surprised at how annoying that was. :-))

Collectives™ on Stack Overflow

Efficient way to perform xml parsing in python

2 Answers 2

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related