Python, Selenium. Google Chrome. Web Scraping. How to navigate between 'tabs' in website

Question

im quite noob in python and right now building up a web scraper in Selenium that would take all URL's for products in the clicked 'tab' on web page. But my code take the URL's from the first 'tab'. Code below. Thank you guys. Im starting to be kind of frustrated lol. Screenshot

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
from lxml import html 


PATH = 'C:\Program Files (x86)\chromedriver.exe'

driver = webdriver.Chrome(PATH)

url = 'https://www.alza.sk/vypredaj-akcia-zlava/e0.htm'

driver.get(url)

driver.find_element_by_xpath('//*[@id="tabs"]/ul/li[2]').click()

links = []

try:
    WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CLASS_NAME, 'blockFilter')))
    link = driver.find_elements_by_xpath("//a[@class='name browsinglink impression-binded']")
    
    for i in link:
        links.append(i.get_attribute('href'))

finally:
    driver.quit()

print(links)

Renan Lopes · Accepted Answer · 2021-03-23 20:39:35Z

3

To select current tab:

current_tab = driver.current_window_handle

To switch between tabs:

driver.switch_to_window(driver.window_handles[1])
driver.switch_to.window(driver.window_handles[-1])

Assuming you have the new tab url as TAB_URL, you should try:

from selenium.webdriver.common.action_chains import ActionChains
action = ActionChains(driver)
action.key_down(Keys.CONTROL).click(TAB_URL).key_up(Keys.CONTROL).perform()

Also, apparently the li doesn't have a click event, are you sure this element you are getting '//*[@id="tabs"]/ul/li[2]' has the aria-selected property set to true or any of these classes: ui-tabs-active ui-state-active?

If not, you should call click on the a tag inside this li.

Then you should increase the timeout parameter of your WebDriverWait to guarantee that the div is loaded.

edited Mar 23, 2021 at 20:39

answered Mar 23, 2021 at 18:46

Renan Lopes

4114 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Ladislaff Over a year ago

Hi thank you. But those are not tabs in google chrome but tabs element specificly in web site. Im trying your method from what i understand after clicking the button it should store the newly formed page with driver.current_window_handle but its not working. Its giving me the out of the range error.

Ladislaff Over a year ago

yes the element have aria-selected = true. I can actually see driver clicking on that tab and loading the product in that tab but when i try to scrape the href's of products with a loop its giving me hrfe's from first tab lol. im wondering if there is some other element above the 'product' elements that a have to specifly select with xpath so it is loading the correct href's

Ladislaff Over a year ago

You mean something like this? It's not working.

try:     WebDriverWait(driver, 10).until(     EC.presence_of_element_located((By.CLASS_NAME, 'browsingitemcontainer')))                                                                                                                            link = driver.find_elements_by_xpath("//a[@class='name browsinglink impression-binded']")          for i in link:         links.append(i.get_attribute('href'))  finally:     driver.quit()

Renan Lopes Over a year ago

Let us continue this discussion in chat.

Collectives™ on Stack Overflow

Python, Selenium. Google Chrome. Web Scraping. How to navigate between 'tabs' in website

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related