Parsing HTML with C++ (using Qt preferably)

Question

I'm trying to parse some HTML with C++ to extract all urls from the HTML (the urls can be inside the href and src attributes).

I tried to use Webkit to do the heavy work for me but for some reason when I load a frame with HTML the generated document is all wrong (if I make Webkit get the page from the web the generated document is just fine but Webkit also downloads all images, styles, and scripts and I don't want that)

Here is what I tried to do:

frame->setHtml(HTML);
QWebElement document = frame->documentElement();
QList<QWebElement> imgs = document.findAll("a"); // Doesn't find all links
QList<QWebElement> imgs = document.findAll("img"); // Doesn't find all images
QList<QWebElement> imgs = document.findAll("script");// Doesn't find all scripts
qDebug() << document.toInnerXml(); // Print a completely messed-up document with several missing elements

What am I doing wrong? Is there an easy way to parse HTML with Qt? (Or some other lightweight library)

1. What "generated document"? 2. What do you mean by "all wrong"? 3. What is the expected behavior? 4. What is the actual behavior? — Billy ONeal
– Billy ONeal, Commented May 22, 2011 at 5:48
@Billy ONeal - When I load the frame with HTML the document structure inside the frame is missing several elements. (this does not happen if I load the page from the web using page->load(url)). — Raphael
– Raphael, Commented May 22, 2011 at 5:52
@ Billy ONeal - When I print the loaded document I can see that it has just some elements of the original HTML. If you put this code in a simple program, compile it you'll see what I'm talking about. — Raphael
– Raphael, Commented May 22, 2011 at 5:55

snoofkin · Accepted Answer · 2011-05-22 08:18:33Z

2

You can always use XPath expressions to make your parsing life easier, take a look at this for instance.

or you can do something like this

QWebView* view = new QWebView(parent);
view.load(QUrl("http://www.your_site.com"));
QWebElementCollection elements = view.page().mainFrame().findAllElements("a");

answered May 22, 2011 at 8:18

snoofkin

8,91514 gold badges52 silver badges89 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Raphael Over a year ago

This only works if the HTML was loaded from the web. If I load the HTML manually it will break on the malformed tags that are present on 90% of the websites.

Collectives™ on Stack Overflow

Parsing HTML with C++ (using Qt preferably)

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related