ElementNotVisibleException Python/Selenium error for next page button click

Question

I am trying to figure out how to use selenium to perform a next page click on a news release page. Here is my code that will go to the correct website and perform a search to obtain the correct news release articles topic page. This site is configured so that to see every news release on page 1 after the search has executed you must also select the More News Results button at the bottom of the page. I am able to get to the full page 1 news section without problem. Here is my code that performs the search and page clicks.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

browser = webdriver.Chrome()
browser.get('http://www.businesswire.com/portal/site/home/')
search_box_element = browser.find_element_by_id('bw-search-input')
search_box_element.clear()
search_box_element.send_keys('biotechnology')
search_box_element.send_keys(Keys.ENTER)
search_box_element_two = browser.find_element_by_id('more-news-results')
search_box_element_two.click()

That part of the code works fine but I want to be able to click on the next button to move to page 2 and then page 3 and so on. Here is the code that I thought would work but it doesn't:

next_page_click_element = browser.find_element_by_class_name("bw-paging-next")
next_page_click_element.click()

This portion of the code throws an error:

selenium.common.exceptions.ElementNotVisibleException: Message: 
element not visible

I have also tried using

next_page_click_element = browser.find_element_by_xpath('//*[@id="more-news-pagination"]/div/div[1]/div/a')

but got the same error message. I have also tried using a wait by adding these lines of codes just before the next_page_click_element section.

element_present = EC.presence_of_element_located((By.ID, "bw-paging-next"))
WebDriverWait(browser, 10).until(element_present)

While this does cause the program to wait it returns this error message:

raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

Any suggestions on how to resolve this issue would be greatly appreciated.

Try to replace (By.ID, "bw-paging-next") with (By.CLASS_NAME, "bw-paging-next") — Andersson
– Andersson, Commented Jun 16, 2017 at 13:57
Hi, Thank you for your response. Unfortunately that does not work. Replacing By.ID with By.CLASS_NAME returns my original Element not visible error message. The change causes no wait period to occur. — user3798654
– user3798654, Commented Jun 16, 2017 at 14:07
Try to execute next_page_click_element.location_once_scrolled_into_view before clicking — Andersson
– Andersson, Commented Jun 16, 2017 at 14:13
Unfortunately that doesn't work either. I still get the original Element not visible error message. — user3798654
– user3798654, Commented Jun 16, 2017 at 14:21

undetected Selenium · Accepted Answer · 2017-06-16 15:40:49Z

1

Here is the Answer to your Question:

A few words about the solution-

As the Page 2,Page 3,Next etc elements are at the bottom of the page, you have to scroll down to bring those elements within the Viewport.
Once you scroll down you may consider to induce some ExplicitWait for the elements to be located properly on the HTML DOM.

Here is your own code with some simple tweaks which will finally scroll to the bottom of the page and click on Next link:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

browser=webdriver.Chrome("C:\\Utility\\BrowserDrivers\\chromedriver.exe")
browser.get('http://www.businesswire.com/portal/site/home/')
search_box_element = browser.find_element_by_id('bw-search-input')
search_box_element.clear()
search_box_element.send_keys('biotechnology')
search_box_element.send_keys(Keys.ENTER)
search_box_element_two = browser.find_element_by_id('more-news-results')
search_box_element_two.click()
last_height = browser.execute_script("return document.body.scrollHeight")
browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
next_page_click_element = WebDriverWait(browser, 10).until(
    EC.presence_of_element_located((By.XPATH, "//div[@id='more-news-pagination']/div/div/div/a[text()='Next']"))
)
next_page_click_element.click()

Let me know if this Answers your Question.

edited Jun 16, 2017 at 15:40

answered Jun 16, 2017 at 15:09

undetected Selenium

194k44 gold badges304 silver badges387 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Andersson Over a year ago

Why do you use time.sleep() right before ExplicitWait? :)

user3798654 Over a year ago

DebanjanB, Thank you for your response. Your solution worked. I really appreciate it. George

user3798654 Over a year ago

Andersson, I think you were on the right track. Thank you for your help.

Collectives™ on Stack Overflow

ElementNotVisibleException Python/Selenium error for next page button click

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related