Rendering Javascript to obtain static HTML in Python

Question

I have a big amount of HTML files which I want to process using BeautifulSoup and generate some statistics. Although, I came across the problem that the HTML files contain scripts that may generate more HTML code which is not being processed. Therefore, I need to render all Javascript into static HTML before proceeding.

I have seen some options such as using Selenium, but it doesn't seem to fit since I don't want to launch a browser (it should be done in background).

Can someone please suggest an appropriate approach to this?

Thanks in advance!

Florent B. · Accepted Answer · 2016-03-17 09:29:14Z

1

Since you need a Javascript engine, using a headless browser is the way to go. Using Selenium web driver with the PhantomJS headless browser is probably your best option:

driver = webdriver.PhantomJS()
driver.get("...")
bs = BeautifulSoup(driver.page_source)

answered Mar 17, 2016 at 9:29

Florent B.

42.7k7 gold badges92 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Rendering Javascript to obtain static HTML in Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related