Full page source (prior to JS rendering) using selenium-python?

Question

I am scraping data from a site with a paginated table (max results 500 with 25 results per page). When I use chrome to "view source" I can see all 500 results, however, once the JS renders in selenium only 25 results show when using driver.page_source.

I have tried passing the cookies and headers off to requests, but that's not reliable and need to stick with selenium. I have also made a janky solution of clicking through the paginator's next button, but there must be a better way!

So how does one capture the full page source prior to JS rendering using selenium with the python bindings?

Update the question with the relevant HTML and your code trials — undetected Selenium
– undetected Selenium, Commented Nov 25, 2018 at 18:16
The page source is irrelevant. This question applies to any scenario in which JS modifies the DOM during rendering. In my current scenario, the JS is hiding page source in JS variables after rendering. I need to capture the page source after it loads from the server and prior to any JS rendering.The only thing I have been able to find is driver.page_source which is obviously returning the source post rendering. — nicholishen
– nicholishen, Commented Nov 25, 2018 at 18:20

pguardiario · Accepted Answer · 2018-11-26 02:44:45Z

1

There might be a simpler way but it turns out you can do all kinds of asynchronous things from the browser including fetch:

def fetch(url):
  return driver.execute_async_script("""
    (async () => {
      let r = await fetch('""" + url + """')
      arguments[0](await r.text())
    })()
  """)

html = fetch('https://stackoverflow.com/')

Same-origin policy will apply.

answered Nov 26, 2018 at 2:44

pguardiario

55.2k21 gold badges130 silver badges169 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Full page source (prior to JS rendering) using selenium-python?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related