Selenium Javascript generated HTML incomplete, even with document.body.innerHTML [duplicate]

Question

I am trying to get the HTML from a webpage that is modified by javascript after/while loading. I have followed the instructions in this tutorial. and I'm using command like this in my Python code after initially loading the page

html = browser.execute_script("return document.body.innerHTML")

While this seems to get all of the HTML Selenium elements on other pages I've tried this on, it does not seem to work on this page. If you open that page and inspect the HTML you can see all of the HTML has loaded. I want to capture the elements below, but can't. By the way WebDriverWait doesn't seem to work either...

<b>
  <a href="contractor_list.asp?alpha=A">A&nbsp;</a>
  <a href="contractor_list.asp?alpha=B">B&nbsp;</a>
  <a href="contractor_list.asp?alpha=C">C&nbsp;</a>
  <a href="contractor_list.asp?alpha=D">D&nbsp;</a>
  ...

How can I get Selenium to expose those elements to me so I can access them? Or, should I be using a different tool for this?

Thanks for the link to the other SO question related to this. I tried to find a related question prior to posting, but at that time didn't know I should be using "iframe" as one of my search keywords. — Jed
– Jed, Commented Sep 30, 2017 at 2:45

B. Fleming · Accepted Answer · 2017-09-29 23:59:42Z

0

Those elements are contained within an iframe element, which is why you're unable to interact with them. Try running the script on the iframe's source if possible.

answered Sep 29, 2017 at 23:59

B. Fleming

7,2501 gold badge21 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Jed Over a year ago

How do I explicitly run the script on that element to expose the rest of the elements?

B. Fleming Over a year ago

I don't think that's possible, actually. I recall using Selenium once in the past and ran into this exact same issue a few times. Unfortunately you'll just have to inspect the element manually, find the iframe, grab its source, and then visit that specific page and run your Selenium script from there. Perhaps it's possible to automate this process? Unfortunately it's been a few years, so I don't really have any advice for setting up the appropriate automation.

Collectives™ on Stack Overflow

Selenium Javascript generated HTML incomplete, even with document.body.innerHTML [duplicate]

1 Answer 1

2 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Linked

Related