Extract data from multiple page using python selenium/Beautifulsoup

Question

Here is the link from where I want to extract the data table from under SDS tab, There are about 40 pages of data however I'am unable to extract the data after going to (clicking on to next button link) next page, here is the code.

driver = webdriver.Chrome("~/chromedriver.exe")
driver.get('https://www.international-pc.com/product/interfine-629')
soup_seln = BeautifulSoup(driver.page_source, 'lxml')
tables_sds = soup_seln.find("table", {"id": "documentTable-2"}).find("tbody")
tr_sds = tables_sds.find_all("tr")           
tr_sds[0].findAll('td')[0].get_text()                                 
Gives O/P --> 'INTERFINE 629 APPLE GREEN PART A'

AFTER Clicking on next page button link

driver.find_element_by_xpath("//a[@id='documentTable-2_next']").click()
tr_sds[0].findAll('td')[0].get_text()

But this doesn't give me new table output. Any suggestion on how can it be done?

@DebanjanB the table (columns) data that is under SDS tab. — Andre_k
– Andre_k, Commented Jun 10, 2019 at 8:44

undetected Selenium · Accepted Answer · 2019-06-10 11:21:43Z

To extract the required data from the SDS tab of the table you can use the following solution:

Code Block:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException, StaleElementReferenceException

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_argument('disable-infobars')
driver=webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("https://www.international-pc.com/product/interfine-629")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[text()='SDS']"))).click()
my_texts = [my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@class='display dataTable no-footer' and @id='documentTable-2']//tbody//tr//td[position() = 1]")))]
while True:
    try:
        WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.paginate_button.next#documentTable-2_next:not(.disabled)"))).click()
        time.sleep(3)
        my_new_text_elements = [my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@class='display dataTable no-footer' and @id='documentTable-2']//tbody//tr//td[position()=1]")))]
        my_texts.extend(my_new_text_elements)
    except (TimeoutException, StaleElementReferenceException) as e:
        break
print(my_texts)

Console Output:

['INTERFINE 629 APPLE GREEN PART A', 'INTERFINE 629 APPLE GREEN PART A', 'INTERFINE 629 B15 MID BLUE PART A', 'INTERFINE 629 B15 MID BLUE PART A', 'INTERFINE 629 B15 MID BLUE PART A', 'INTERFINE 629 B15 MID BLUE PART A', 'INTERFINE 629 B44 LIGHT GREY BLUE PTA', 'INTERFINE 629 BASE LIGHT PART A', 'INTERFINE 629 BASE LIGHT PART A', 'INTERFINE 629 BASE LIGHT PART A', 'INTERFINE 629 BASE LIGHT PART A', 'INTERFINE 629 BASE LIGHT PART A', 'INTERFINE 629 BASE LIGHT PART A', 'INTERFINE 629 BASE LIGHT PART A', 'INTERFINE 629 BASE MEDIUM PART A', 'INTERFINE 629 BASE MEDIUM PART A', 'INTERFINE 629 BASE MEDIUM PART A', 'INTERFINE 629 BASE MEDIUM PART A', 'INTERFINE 629 BASE RED PART A', 'INTERFINE 629 BASE RED PART A', 'INTERFINE 629 BASE RED PART A', 'INTERFINE 629 BASE RED PART A', 'INTERFINE 629 BASE ULTRA DEEP PART A', 'INTERFINE 629 BASE ULTRA DEEP PART A', 'INTERFINE 629 BASE ULTRA DEEP PART A', 'INTERFINE 629 BASE ULTRA DEEP PART A', 'INTERFINE 629 BASE ULTRA DEEP PART A', 'INTERFINE 629 BASE ULTRA DEEP PART A', 'INTERFINE 629 BASE YELLOW PART A', 'INTERFINE 629 BASE YELLOW PART A', 'INTERFINE 629 BASE YELLOW PART A', 'INTERFINE 629 BASE YELLOW PART A', 'INTERFINE 629 BASE YELLOW PART A', 'INTERFINE 629 BASE YELLOW PART A', 'INTERFINE 629 BLACK PART A', 'INTERFINE 629 BLACK PART A', 'INTERFINE 629 BLACK PART A', 'INTERFINE 629 BLACK PART A', 'INTERFINE 629 BLACK PART A', 'INTERFINE 629 BLACK PART A', 'INTERFINE 629 BLUE GREY PART A', 'INTERFINE 629 BLUE GREY PART A', 'INTERFINE 629 BLUE GREY PART A', 'INTERFINE 629 BLUE GREY PART A', 'INTERFINE 629 BLUE GREY PART A', 'INTERFINE 629 BLUE GREY PART A', 'INTERFINE 629 BLUE GREY PART A', 'INTERFINE 629 BLUE GREY PART A', 'INTERFINE 629 BLUE JADE PART A', 'INTERFINE 629 BLUE JADE PART A', 'INTERFINE 629 BLUE PART A', 'INTERFINE 629 BLUE PART A', 'INTERFINE 629 BLUED WHITE PART A', 'INTERFINE 629 BLUED WHITE PART A', 'INTERFINE 629 BRIGHT RED ORANGE PART A', 'INTERFINE 629 BRIGHT RED ORANGE PART A', 'INTERFINE 629 CAT YELLOW PART A', 'INTERFINE 629 CAT YELLOW PART A', 'INTERFINE 629 CAT YELLOW PART A', 'INTERFINE 629 CATERPILLAR YELLOW PT A', 'INTERFINE 629 CATERPILLAR YELLOW PT A', 'INTERFINE 629 CEMENT GREY PART A', 'INTERFINE 629 CEMENT GREY PART A', 'INTERFINE 629 CLAY BROWN (RAL8003)PT A', 'INTERFINE 629 CLOUD GREY PART A', 'INTERFINE 629 CLOUD GREY PART A', 'INTERFINE 629 CLOUD GREY PART A', 'INTERFINE 629 CLOUD GREY PART A', 'INTERFINE 629 CLOUD GREY PART A', 'INTERFINE 629 CLOUD GREY PART A', 'INTERFINE 629 CORFLOWER PART A', 'INTERFINE 629 CORFLOWER PART A', 'INTERFINE 629 CURING AGENT', 'INTERFINE 629 DARK BLUE PART A', 'INTERFINE 629 DEEP BASE PART A', 'INTERFINE 629 FINISH BLUE GREEN', 'INTERFINE 629 FLOTILLA GRAY PART A', 'INTERFINE 629 FLOTILLA GRAY PART A', 'INTERFINE 629 FLOTILLA GRAY PART A', 'INTERFINE 629 FLOTILLA GRAY PART A', 'INTERFINE 629 FLOTILLA GRAY PART A', 'INTERFINE 629 FLOTILLA GRAY PART A', 'INTERFINE 629 G21 JADE PART A', 'INTERFINE 629 G25 OLIVE PART A', 'INTERFINE 629 GARDENIA PART A', 'INTERFINE 629 GARDENIA PART A', 'INTERFINE 629 GARDENIA PART A', 'INTERFINE 629 GARDENIA PART A', 'INTERFINE 629 GARDENIA PART A', 'INTERFINE 629 GLOAMING PART A', 'INTERFINE 629 GLOAMING PART A', 'INTERFINE 629 GLOAMING PART A', 'INTERFINE 629 GLOAMING PART A', 'INTERFINE 629 GOLDEN TAN PART A', 'INTERFINE 629 GOLDEN TAN PART A', 'INTERFINE 629 GOLDEN YELLOW PART A', 'INTERFINE 629 GOLDEN YELLOW PART A', 'INTERFINE 629 GOLDEN YELLOW PART A', 'INTERFINE 629 GOLDEN YELLOW PART A', 'INTERFINE 629 GOLDEN YELLOW PART A', 'INTERFINE 629 GOLDEN YELLOW PART A', 'INTERFINE 629 GOLDEN YELLOW PART A', 'INTERFINE 629 GOLDEN YELLOW PART A', 'INTERFINE 629 GOLDEN YELLOW PART A', 'INTERFINE 629 GOLDEN YELLOW PART A', 'INTERFINE 629 GOLDEN YELLOW PART A', 'INTERFINE 629 GOLDEN YELLOW PART A', 'INTERFINE 629 GOLDEN YELLOW PART A', 'INTERFINE 629 GOLDEN YELLOW PART A', 'INTERFINE 629 GRAPHITE GREY PART A', 'INTERFINE 629 GRAPHITE GREY PART A', 'INTERFINE 629 GRASS GREEN(RAL6010)PT A', 'INTERFINE 629 GREEN PART A', 'INTERFINE 629 GREEN PART A', 'INTERFINE 629 GREY PART A', 'INTERFINE 629 GREY PART A', 'INTERFINE 629 HALIBURTON RED PART A', 'INTERFINE 629 HS FINISH AUTUMN CLOUD', 'INTERFINE 629 HS FINISH BLUE BELL BASE', 'INTERFINE 629 HS FINISH CREAM BASE', 'INTERFINE 629 HS FINISH FISHER GREEN BASE', 'INTERFINE 629 HS FINISH FISHER GREY BASE', 'INTERFINE 629 HS FINISH GLOAMING BASE', 'INTERFINE 629 HS FINISH GOLDEN YELLOW BASE', 'INTERFINE 629 HS FINISH GREY BASE', 'INTERFINE 629 HS FINISH MARBLE GREY BASE', 'INTERFINE 629 HS FINISH MIST GREY BASE', 'INTERFINE 629 HS FINISH MOCHA BASE', 'INTERFINE 629 HS FINISH POPPY RED BASE', 'INTERFINE 629 HS FINISH POWDER BLUE BASE', 'INTERFINE 629 HS FINISH TANGERINE', 'INTERFINE 629 HS FINISH YELLOW BASE', 'INTERFINE 629 HS LIGHT GREY BASE', 'INTERFINE 629 HS LIGHT GREY BASE', 'INTERFINE 629 HS LIGHT GREY BASE', 'INTERFINE 629 HS LIGHT GREY BASE', 'INTERFINE 629 HS YELLOW BASE', 'INTERFINE 629 JADE PART A', 'INTERFINE 629 JADE PART A', 'INTERFINE 629 JADE PART A', 'INTERFINE 629 JET BLACK PART A', 'INTERFINE 629 JET BLACK PART A', 'INTERFINE 629 JET BLACK PART A', 'INTERFINE 629 JET BLACK PART A', 'INTERFINE 629 JET BLACK PART A', 'INTERFINE 629 JET BLACK PART A', 'INTERFINE 629 LIGHT BOX GREY PART A', 'INTERFINE 629 LIGHT BOX GREY PART A', 'INTERFINE 629 LIGHT BOX GREY PART A', 'INTERFINE 629 LIGHT BOX GREY PART A', 'INTERFINE 629 LIGHT BOX GREY PART A', 'INTERFINE 629 LIGHT BOX GREY PART A', 'INTERFINE 629 LIGHT GREY BLUE PART A', 'INTERFINE 629 LIGHT GREY BLUE PART A', 'INTERFINE 629 LIGHT GREY PART A', 'INTERFINE 629 LIGHT GREY PART A', 'INTERFINE 629 LIGHT IVORY PART A', 'INTERFINE 629 LIGHTBOX GREY PART A', 'INTERFINE 629 LIGHTBOX GREY PART A', 'INTERFINE 629 MAUVE PART A', 'INTERFINE 629 MIST GREEN PART A', 'INTERFINE 629 MIST GREEN PART A', 'INTERFINE 629 MIST GREY PART A', 'INTERFINE 629 MIST GREY PART A', 'INTERFINE 629 N14 WHITE PART A', 'INTERFINE 629 N14 WHITE PART A', 'INTERFINE 629 N14 WHITE PART A', 'INTERFINE 629 N35 LIGHT GREY PART A', 'INTERFINE 629 N35 LIGHT GREY PART A', 'INTERFINE 629 N35 LIGHT GREY PART A', 'INTERFINE 629 N35 LIGHT GREY PART A', 'INTERFINE 629 N52 MID GREY PART A', 'INTERFINE 629 N52 MID GREY PART A', 'INTERFINE 629 N52 MID GREY PART A', 'INTERFINE 629 N52 MID GREY PART A', 'INTERFINE 629 NASSCO WHITE PART A', 'INTERFINE 629 NASSCO WHITE PART A', 'INTERFINE 629 OATMEAL PART A', 'INTERFINE 629 OFF WHITE PART A', 'INTERFINE 629 OFF WHITE PART A', 'INTERFINE 629 OFF WHITE PART A', 'INTERFINE 629 OFF WHITE PART A', 'INTERFINE 629 OFF WHITE PART A', 'INTERFINE 629 OYSTER PART A', 'INTERFINE 629 OYSTER PART A', 'INTERFINE 629 OYSTER WHITE PART A', 'INTERFINE 629 PARAKEET PART A', 'INTERFINE 629 PARAKEET PART A', 'INTERFINE 629 PART B', 'INTERFINE 629 PART B', 'INTERFINE 629 PART B', 'INTERFINE 629 PART B', 'INTERFINE 629 PART B', 'INTERFINE 629 PART B', 'INTERFINE 629 PART B', 'INTERFINE 629 PART B', 'INTERFINE 629 PART B', 'INTERFINE 629 PART B', 'INTERFINE 629 PART B', 'INTERFINE 629 PART B', 'INTERFINE 629 PART B', 'INTERFINE 629 PART B', 'INTERFINE 629 PART B', 'INTERFINE 629 PART B', 'INTERFINE 629 PART B', 'INTERFINE 629 PART B', 'INTERFINE 629 POWDER BLUE PART A', 'INTERFINE 629 POWDER BLUE PART A', 'INTERFINE 629 RAINFOREST GREEN PART A', 'INTERFINE 629 RAL1013OYSTER WHITE PT A', 'INTERFINE 629 RAL1013OYSTER WHITE PT A', 'INTERFINE 629 RAL1021 YELLOW PART A', 'INTERFINE 629 RAL1021 YELLOW PART A', 'INTERFINE 629 RAL1023 TRAFF.YELLOW PAR', 'INTERFINE 629 RAL1023 TRAFF.YELLOW PTA', 'INTERFINE 629 RAL1023 TRAFF.YELLOW PTA', 'INTERFINE 629 RAL1023TRAFFICYELLOW PTA', 'INTERFINE 629 RAL1023TRAFFICYELLOW PTA', 'INTERFINE 629 RAL1023TRAFFICYELLOW PTA', 'INTERFINE 629 RAL1023TRAFFICYELLOW PTA', 'INTERFINE 629 RAL2004 PURE ORANGE PT A', 'INTERFINE 629 RAL2004 PURE ORANGE PT A', 'INTERFINE 629 RAL2004 PURE ORANGE PTA', 'INTERFINE 629 RAL2004 PURE ORANGE PTA', 'INTERFINE 629 RAL3000 FLAME RED PART A', 'INTERFINE 629 RAL3000 FLAME RED PART A', 'INTERFINE 629 RAL3000 FLAME RED PT A', 'INTERFINE 629 RAL3000 FLAME RED PT A', 'INTERFINE 629 RAL3002 CARMINE RED PT A', 'INTERFINE 629 RAL3020 TRAFFIC RED PT A', 'INTERFINE 629 RAL3020 TRAFFIC RED PT A', 'INTERFINE 629 RAL3020 TRAFFIC RED PT A', 'INTERFINE 629 RAL3020 TRAFFIC RED PT A', 'INTERFINE 629 RAL3020 TRAFFIC RED PTA', 'INTERFINE 629 RAL3020 TRAFFIC RED PTA', 'INTERFINE 629 RAL3020 TRAFFIC RED PTA', 'INTERFINE 629 RAL3020 TRAFFIC RED PTA', 'INTERFINE 629 RAL3020 TRAFFIC RED PTA', 'INTERFINE 629 RAL3020 TRAFFIC RED PTA', 'INTERFINE 629 RAL5002 ULTRAMAR BLUE PA', 'INTERFINE 629 RAL5002 ULTRAMAR BLUE PA', 'INTERFINE 629 RAL5003 SAPPHIRE BLUE PA', 'INTERFINE 629 RAL5003 SAPPHIRE BLUE PA', 'INTERFINE 629 RAL5009 AZURE BLUE PT A', 'INTERFINE 629 RAL5009 AZURE BLUE PT A', 'INTERFINE 629 RAL5009 AZURE BLUE PT A', 'INTERFINE 629 RAL5009 AZURE BLUE PT A', 'INTERFINE 629 RAL5009 AZURE BLUE PT A', 'INTERFINE 629 RAL5009 AZURE BLUE PT A', 'INTERFINE 629 RAL5015 SKY BLUE PART A', 'INTERFINE 629 RAL5015 SKY BLUE PART A', 'INTERFINE 629 RAL5015 SKY BLUE PT A', 'INTERFINE 629 RAL5015 SKY BLUE PT A', 'INTERFINE 629 RAL6002 LEAF GREEN PT A', 'INTERFINE 629 RAL6002 LEAF GREEN PT A', 'INTERFINE 629 RAL6002 LEAF GREEN PT A', 'INTERFINE 629 RAL6002 LEAF GREEN PT A', 'INTERFINE 629 RAL6002 LEAF GREEN PT A', 'INTERFINE 629 RAL6002 LEAF GREEN PT A', 'INTERFINE 629 RAL6011 RESEDA GREEN PTA', 'INTERFINE 629 RAL6017 MAY GREEN PART A', 'INTERFINE 629 RAL6017 MAY GREEN PART A', 'INTERFINE 629 RAL6018 YELLOW GREEN PAR', 'INTERFINE 629 RAL6018 YELLOW GREEN PAR', 'INTERFINE 629 RAL6021 PALE GREEN PT A', 'INTERFINE 629 RAL6021 PALE GREEN PT A', 'INTERFINE 629 RAL6021 PALE GREEN PT A', 'INTERFINE 629 RAL6021 PALE GREEN PT A', 'INTERFINE 629 RAL6021 PALE GREEN PT A', 'INTERFINE 629 RAL6021 PALE GREEN PT A', 'INTERFINE 629 RAL6021 PALE GREEN PT A', 'INTERFINE 629 RAL6028 PINE GREEN PT A', 'INTERFINE 629 RAL6028 PINE GREEN PT A', 'INTERFINE 629 RAL7005 MOUSE GREY PT A', 'INTERFINE 629 RAL7005 MOUSE GREY PT A', 'INTERFINE 629 RAL7030 STONE GREY PTA', 'INTERFINE 629 RAL7030 STONE GREY PTA', 'INTERFINE 629 RAL7032 GREY PART A', 'INTERFINE 629 RAL7032 GREY PART A', 'INTERFINE 629 RAL7032 GREY PART A', 'INTERFINE 629 RAL7032 GREY PART A', 'INTERFINE 629 RAL7032 GREY PART A', 'INTERFINE 629 RAL7032 GREY PART A', 'INTERFINE 629 RAL7032 GREY PART A', 'INTERFINE 629 RAL7032 GREY PART A', 'INTERFINE 629 RAL7032 PEBBLE GREY PT A', 'INTERFINE 629 RAL7035 LIGHT GREY PT A', 'INTERFINE 629 RAL7035 LIGHT GREY PT A', 'INTERFINE 629 RAL7035 LIGHT GREY PTA', 'INTERFINE 629 RAL7035 LIGHT GREY PTA', 'INTERFINE 629 RAL7036 PLATINUMGREY PTA', 'INTERFINE 629 RAL7036 PLATINUMGREY PTA', 'INTERFINE 629 RAL7037 DUSTY GREY PARTA', 'INTERFINE 629 RAL7037 DUSTY GREY PARTA', 'INTERFINE 629 RAL7037 DUSTY GREY PT A', 'INTERFINE 629 RAL7037 DUSTY GREY PT A', 'INTERFINE 629 RAL7037 DUSTY GREY PT A', 'INTERFINE 629 RAL7037 DUSTY GREY PT A', 'INTERFINE 629 RAL7037 DUSTY GREY PT A', 'INTERFINE 629 RAL7037 DUSTY GREY PT A', 'INTERFINE 629 RAL7037 DUSTY GREY PT A', 'INTERFINE 629 RAL7037 DUSTY GREY PT A', 'INTERFINE 629 RAL7038 AGATE GREY PT A', 'INTERFINE 629 RAL7038 AGATE GREY PT A', 'INTERFINE 629 RAL7038 AGATE GREY PT A', 'INTERFINE 629 RAL7038 AGATE GREY PT A', 'INTERFINE 629 RAL7038 AGATE GREY PT A', 'INTERFINE 629 RAL7038 AGATE GREY PT A', 'INTERFINE 629 RAL8001 OCHRE BROWN PT A', 'INTERFINE 629 RAL8001 OCHRE BROWN PT A', 'INTERFINE 629 RAL9002 GREY WHITE PT A', 'INTERFINE 629 RAL9002 GREY WHITE PT A', 'INTERFINE 629 RAL9002 GREY WHITE PT A', 'INTERFINE 629 RAL9002 GREY WHITE PT A', 'INTERFINE 629 RAL9010 PURE WHITE PT A', 'INTERFINE 629 RAL9010 PURE WHITE PT A', 'INTERFINE 629 RAL9010 PURE WHITE PT A', 'INTERFINE 629 RAL9010 PURE WHITE PT A', 'INTERFINE 629 RAL9010 PURE WHITE PT A', 'INTERFINE 629 RAL9010 PURE WHITE PT A', 'INTERFINE 629 RAL9016 TRAFFIC WHITE PA', 'INTERFINE 629 RAL9016 TRAFFIC WHITE PA', 'INTERFINE 629 RIVERGUM PART A', 'INTERFINE 629 RIVERGUM PART A', 'INTERFINE 629 RIVERGUM PART A', 'INTERFINE 629 RIVERGUM PART A', 'INTERFINE 629 ROSE PINK PART A', 'INTERFINE 629 ROSE PINK PART A', 'INTERFINE 629 SAND PART A', 'INTERFINE 629 SAND PART A', 'INTERFINE 629 SIGNAL RED PART A', 'INTERFINE 629 SIGNAL RED PART A', 'INTERFINE 629 SIGNAL WHITE PART A', 'INTERFINE 629 STORM GREY PART A', 'INTERFINE 629 STORM GREY PART A', 'INTERFINE 629 STORM GREY PART A', 'INTERFINE 629 STORM GREY PART A', 'INTERFINE 629 SUNFLOWER PART A', 'INTERFINE 629 SUNFLOWER PART A', 'INTERFINE 629 SUNFLOWER PART A', 'INTERFINE 629 SUNFLOWER PART A', 'INTERFINE 629 TANGERINE PART A', 'INTERFINE 629 TANGERINE PART A', 'INTERFINE 629 TANGERINE PART A', 'INTERFINE 629 TANGERINE PART A', 'INTERFINE 629 TANGERINE PART A', 'INTERFINE 629 TANGERINE PART A', 'INTERFINE 629 TELEGREY PART A', 'INTERFINE 629 TRAFFIC BLACK PART A', 'INTERFINE 629 TRAFFIC YELLOW PART A', 'INTERFINE 629 VIVID YELLOW PART A', 'INTERFINE 629 WARATAH PART A', 'INTERFINE 629 WARATAH PART A', 'INTERFINE 629 WHITE PART A', 'INTERFINE 629 WHITE PART A', 'INTERFINE 629 WHITE PART A', 'INTERFINE 629 WHITE PART A', 'INTERFINE 629 WHITE PART A', 'INTERFINE 629 WHITE PART A', 'INTERFINE 629 WHITE PART A', 'INTERFINE 629 WHITE PART A', 'INTERFINE 629 WHITE PART A', 'INTERFINE 629 WHITE PART A', 'INTERFINE 629 WHITE PART A', 'INTERFINE 629 WHITE PART A', 'INTERFINE 629 WHITE PART A', 'INTERFINE 629 WHITE PART A', 'INTERFINE 629 Y12 WATTLE PART A', 'INTERFINE 629 Y12 WATTLE PART A', 'INTERFINE 629 Y12 WATTLE PART A', 'INTERFINE 629 Y12 WATTLE PART A', 'INTERFINE 629 Y14 GOLDEN YELLOW PART A', 'INTERFINE 629 Y14 GOLDEN YELLOW PART A', 'INTERFINE 629 Y14 GOLDEN YELLOW PART A', 'INTERFINE 629 Y14 GOLDEN YELLOW PART A', 'INTERFINE 629 Y14 GOLDEN YELLOW PART A', 'INTERFINE 629 Y14 GOLDEN YELLOW PART A', 'INTERFINE 629 Y14 GOLDEN YELLOW PART A', 'INTERFINE 629 Y15 SUNFLOWER PART A', 'INTERFINE 629 Y34 CREAM PART A', 'INTERFINE 629 Y44 SAND PART A', 'INTERFINE 629 YELLOW GREY PART A', 'INTERFINE 629 YELLOW GREY PART A', 'INTERFINE 629 YELLOW PART A', 'INTERFINE 629 YELLOW PART A', 'INTERFINE 629 YELLOW PART A', 'INTERFINE 629 YELLOW PART A', 'INTERFINE 629 YELLOW PART A', 'INTERFINE 629 YELLOW PART A', 'INTERFINE 629 ZINC YELLOW PART A', 'INTERFINE 629 ZINC YELLOW PART A']

Its working as per requirement now.Thanks. If you could please let me know any link from where I can learn these things, it will be of a great help.Thanks
@deepesh In short, as a thumb rule, when you need to read any attribute e.g. text, innerHTML, etc of an element you have to use visibility_of_all_elements_located() where as if you want to click() any element you have to use element_to_be_clickable()
In that case I used to use find_elements_by_xpath() when working with selenium. Thanks once again for your input and time.
Please don't assume anything to be present/visible/interactable automatically. You have all the Selenium based methods at your disposal which you need to utilize wisely.
Yes, with this current going on project , now I have came across new functions/ methods of selenium, which now I really need to learn them in depth..appreciate all of your help

Jainil Patel · Accepted Answer · 2019-06-10 08:10:35Z

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time
import random
from bs4 import BeautifulSoup
driver = webdriver.Chrome()
driver.get('https://www.international-pc.com/product/interfine-629')
soup_seln = BeautifulSoup(driver.page_source, 'lxml')
time.sleep(5)
x=driver.find_element_by_xpath("//*[@id='cookie-notification']/div/article/div[3]/a[1]")
x.click()


tables_sds = soup_seln.find("table", {"id": "documentTable-2"}).find("tbody")
tr_sds = tables_sds.find_all("tr")
for i in range(len(tr_sds)):
    print('td')[0].get_text()+"                   "+tr_sds[i].findAll('td')[1].get_text()+"                           "+tr_sds[i].findAll('td')[2].get_text())

while True:
    time.sleep(5)
    x=driver.find_element_by_xpath("//*[@id='documentTable-2_next']")
    x.click()
    tables_sds = soup_seln.find("table", {"id": "documentTable-2"}).find("tbody")
    tr_sds = tables_sds.find_all("tr")
    for i in range(len(tr_sds)):
        print(tr_sds[i].findAll('td')[0].get_text()+"                   "+tr_sds[i].findAll('td')[1].get_text()+"                           "+tr_sds[i].findAll('td')[2].get_text())

don't need to click on agree, but you have 5 seconds once and you have to click SDS button manually once. if you find any difficulty running this code, i will help you, my sample results is:

INTERFINE 629 APPLE GREEN PART A
INTERFINE 629 APPLE GREEN PART A
INTERFINE 629 B15 MID BLUE PART A
INTERFINE 629 B15 MID BLUE PART A
INTERFINE 629 B15 MID BLUE PART A
INTERFINE 629 B15 MID BLUE PART A
INTERFINE 629 B44 LIGHT GREY BLUE PTA
INTERFINE 629 BASE LIGHT PART A
INTERFINE 629 BASE LIGHT PART A
INTERFINE 629 BASE LIGHT PART A
INTERFINE 629 APPLE GREEN PART A                   Chinese                           China
INTERFINE 629 APPLE GREEN PART A                   English                           China
INTERFINE 629 B15 MID BLUE PART A                   Japanese                           Japan
INTERFINE 629 B15 MID BLUE PART A                   English                           Japan
INTERFINE 629 B15 MID BLUE PART A                   Chinese                           Singapore
INTERFINE 629 B15 MID BLUE PART A                   English                           Singapore
INTERFINE 629 B44 LIGHT GREY BLUE PTA                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Australia
INTERFINE 629 BASE LIGHT PART A                   Malayalam                           Malaysia
INTERFINE 629 APPLE GREEN PART A                   Chinese                           China
INTERFINE 629 APPLE GREEN PART A                   English                           China
INTERFINE 629 B15 MID BLUE PART A                   Japanese                           Japan
INTERFINE 629 B15 MID BLUE PART A                   English                           Japan
INTERFINE 629 B15 MID BLUE PART A                   Chinese                           Singapore
INTERFINE 629 B15 MID BLUE PART A                   English                           Singapore
INTERFINE 629 B44 LIGHT GREY BLUE PTA                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Australia
INTERFINE 629 BASE LIGHT PART A                   Malayalam                           Malaysia
INTERFINE 629 APPLE GREEN PART A                   Chinese                           China
INTERFINE 629 APPLE GREEN PART A                   English                           China
INTERFINE 629 B15 MID BLUE PART A                   Japanese                           Japan
INTERFINE 629 B15 MID BLUE PART A                   English                           Japan
INTERFINE 629 B15 MID BLUE PART A                   Chinese                           Singapore
INTERFINE 629 B15 MID BLUE PART A                   English                           Singapore
INTERFINE 629 B44 LIGHT GREY BLUE PTA                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Australia
INTERFINE 629 BASE LIGHT PART A                   Malayalam                           Malaysia
INTERFINE 629 APPLE GREEN PART A                   Chinese                           China
INTERFINE 629 APPLE GREEN PART A                   English                           China
INTERFINE 629 B15 MID BLUE PART A                   Japanese                           Japan
INTERFINE 629 B15 MID BLUE PART A                   English                           Japan
INTERFINE 629 B15 MID BLUE PART A                   Chinese                           Singapore
INTERFINE 629 B15 MID BLUE PART A                   English                           Singapore
INTERFINE 629 B44 LIGHT GREY BLUE PTA                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Australia
INTERFINE 629 BASE LIGHT PART A                   Malayalam                           Malaysia
INTERFINE 629 APPLE GREEN PART A                   Chinese                           China
INTERFINE 629 APPLE GREEN PART A                   English                           China
INTERFINE 629 B15 MID BLUE PART A                   Japanese                           Japan
INTERFINE 629 B15 MID BLUE PART A                   English                           Japan
INTERFINE 629 B15 MID BLUE PART A                   Chinese                           Singapore
INTERFINE 629 B15 MID BLUE PART A                   English                           Singapore
INTERFINE 629 B44 LIGHT GREY BLUE PTA                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Australia
INTERFINE 629 BASE LIGHT PART A                   Malayalam                           Malaysia
INTERFINE 629 APPLE GREEN PART A                   Chinese                           China
INTERFINE 629 APPLE GREEN PART A                   English                           China
INTERFINE 629 B15 MID BLUE PART A                   Japanese                           Japan
INTERFINE 629 B15 MID BLUE PART A                   English                           Japan
INTERFINE 629 B15 MID BLUE PART A                   Chinese                           Singapore
INTERFINE 629 B15 MID BLUE PART A                   English                           Singapore
INTERFINE 629 B44 LIGHT GREY BLUE PTA                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Australia
INTERFINE 629 BASE LIGHT PART A                   Malayalam                           Malaysia
INTERFINE 629 APPLE GREEN PART A                   Chinese                           China
INTERFINE 629 APPLE GREEN PART A                   English                           China
INTERFINE 629 B15 MID BLUE PART A                   Japanese                           Japan
INTERFINE 629 B15 MID BLUE PART A                   English                           Japan
INTERFINE 629 B15 MID BLUE PART A                   Chinese                           Singapore
INTERFINE 629 B15 MID BLUE PART A                   English                           Singapore
INTERFINE 629 B44 LIGHT GREY BLUE PTA                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Australia
INTERFINE 629 BASE LIGHT PART A                   Malayalam                           Malaysia
INTERFINE 629 APPLE GREEN PART A                   Chinese                           China
INTERFINE 629 APPLE GREEN PART A                   English                           China
INTERFINE 629 B15 MID BLUE PART A                   Japanese                           Japan
INTERFINE 629 B15 MID BLUE PART A                   English                           Japan
INTERFINE 629 B15 MID BLUE PART A                   Chinese                           Singapore
INTERFINE 629 B15 MID BLUE PART A                   English                           Singapore
INTERFINE 629 B44 LIGHT GREY BLUE PTA                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Australia
INTERFINE 629 BASE LIGHT PART A                   Malayalam                           Malaysia
INTERFINE 629 APPLE GREEN PART A                   Chinese                           China
INTERFINE 629 APPLE GREEN PART A                   English                           China
INTERFINE 629 B15 MID BLUE PART A                   Japanese                           Japan
INTERFINE 629 B15 MID BLUE PART A                   English                           Japan
INTERFINE 629 B15 MID BLUE PART A                   Chinese                           Singapore
INTERFINE 629 B15 MID BLUE PART A                   English                           Singapore
INTERFINE 629 B44 LIGHT GREY BLUE PTA                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Thailand
INTERFINE 629 BASE LIGHT PART A                   English                           Australia

@Janil Patel, please note that the values that you have printed are not changing, in fact they are repeating after every 10th element.

Collectives™ on Stack Overflow

Extract data from multiple page using python selenium/Beautifulsoup

2 Answers 2

9 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

9 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related