Python: Is there a way to get HTML that was dynamically created by Javascript?

Question

As far as I can tell, this is the case for LyricWikia. The lyrics (example) can be accessed from the browser, but can't be found in the source code (can be opened with CTRL + U in most browsers) or reading the contents of the site with Python:

from urllib.request import urlopen

URL = 'http://lyrics.wikia.com/Billy_Joel:Piano_Man'

r = urlopen(URL).read().decode('utf-8')

And the test:

>>> 'Now John at the bar is a friend of mine' in r
False
>>> 'John' in r
False

But when you select and look at the source code of the box in which the lyrics are displayed, you can see that there is: <div class="lyricbox">[...]</div>

Is there a way to get the contents of that div-element with Python?

Alexander Gessler · Accepted Answer · 2014-08-25 16:18:55Z

2

You can try Ghost.py, which is essentially Phantom.js for Python. It embeds WebKit and is thus able to execute the JavaScript on the page as if you had navigated to the page manually. It then gives you access to the DOM structure.

answered Aug 25, 2014 at 16:18

Alexander Gessler

46.9k7 gold badges86 silver badges124 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Feign Over a year ago

This is exactly what I've been looking for, coincidentally. Answered my question before I even asked it :)

Collectives™ on Stack Overflow

Python: Is there a way to get HTML that was dynamically created by Javascript?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related