Selenium returns different html source than viewed in browser

Question

Im trying to use Selenium to load next page with results by clicking Load More button from this site. However the source code of the html page loaded by selenium does not show(load) actual products which one can see when browsing. Here is my code:

from selenium import webdriver      
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import os
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

#browser = webdriver.Firefox()#Chrome('./chromedriver.exe')
URL = "https://thekrazycouponlady.com/coupons-for/costco"
PATIENCE_TIME = 60
LOAD_MORE_BUTTON_XPATH = '//button[@class = "kcl-btn ng-scope"]/span' 
caps = DesiredCapabilities.PHANTOMJS
# driver = webdriver.Chrome(r'C:\Python3\selenium\webdriver\chromedriver_win32\chromedriver.exe')
caps["phantomjs.page.settings.userAgent"] = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36"
driver = webdriver.PhantomJS(r'C:\Python3\selenium\webdriver\phantomjs-2.1.1-windows\bin\phantomjs.exe',service_log_path=os.path.devnull,desired_capabilities=caps)
driver.get(URL)

while True:
    try:
        time.sleep(20)
        html = driver.page_source.encode('utf-8')
        print(html)
        loadMoreButton = driver.find_element_by_xpath(LOAD_MORE_BUTTON_XPATH)


        loadMoreButton.click()

    except Exception as e:
        print (e)
        break
print ("Complete")

driver.quit()

Not sure if I can attach sample html file here for reference. Anyway, what is the problem and how do I load exactly the same page with selenium as i do via browser?

What do you mean by the code returned by Selenium?

Guy
– Guy

2018-12-10 14:13:36 +00:00
Commented Dec 10, 2018 at 14:13 — Guy
– Guy, Commented Dec 10, 2018 at 14:13
@Guy The source code of the html page loaded by selenium.

JBJ
– JBJ

2018-12-10 14:21:17 +00:00
Commented Dec 10, 2018 at 14:21 — JBJ
– JBJ, Commented Dec 10, 2018 at 14:21

JJJ · Accepted Answer · 2018-12-10 16:15:40Z

2

It might be due to the use of PhantomJS, it isn't maintained any more and deprecated from Selenium 3.8.1. Use Chrome headless instead.

options = Options()
options.headless = True
driver = webdriver.Chrome(CHROMEDRIVER_PATH, chrome_options=options)

edited Dec 10, 2018 at 16:15

JJJ

33.2k20 gold badges95 silver badges103 bronze badges

answered Dec 10, 2018 at 14:33

Guy

51.2k10 gold badges49 silver badges96 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

JBJ Over a year ago

Thanks. That returned the right content. Though now Im getting 2 other errors - </button> is not clickable at point (391, 579) and Element not interactable. I see some banner over the page. Can this be a problem?

Guy Over a year ago

@BillyJhon Yes. If the banner disappears automatically you can wait for it to disappear, have a look at explicit wait, something like WebDriverWait(driver, 10).until(EC.invisibility_of_element_located((By.ID, "some id")). Otherwise you need to close it.

Collectives™ on Stack Overflow

Selenium returns different html source than viewed in browser

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related