Selenium: How to parse through a code after using selenium to click a dropdown

Question

I'm trying to webscrape trough this webpage https://www.sigmaaldrich.com/. Up to now I have achieved for the code to use the requests method to use the search bar. After that, I want to look for the different prices of the compounds. The html code that includes the prices is not visible until the Price dropdown has been clicked. I have achieved that by using selenium to click all the dropdowns with the desired class. But after that, I do not know how to get the html code of the webpage that is generated after clicking the dropdowns and where the price is placed.

Here's my code so far:

import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from time import sleep

#get the desired search terms by imput

 name=input("Reagent: ")
 CAS=input("CAS: ")

  #search using the name of the compound

  data_name= {'term':name, 'interface':'Product%20Name', 'N':'0+',
       'mode':'mode%20matchpartialmax', 'lang':'es','region':'ES',
        'focus':'product', 'N':'0%20220003048%20219853286%20219853112'}
   #search using the CAS of the compound

   data_CAS={'term':CAS, 'interface':'CAS%20No.', 'N':'0','mode':'partialmax',
        'lang':'es', 'region':'ES', 'focus':'product'}
#get the link of the name search
 r=requests.post("https://www.sigmaaldrich.com/catalog/search/",    params=data_name.items())

 #get the link of the CAS search
n=requests.post("https://www.sigmaaldrich.com/catalog/search/",    params=data_CAS.items())

#use selenium to click in the dropdown(only for the name search)

driver=webdriver.Chrome(executable_path=r"C:\webdrivers\chromedriver.exe")
driver.get(r.url)
dropdown=driver.find_elements_by_class_name("expandArrow")
for arrow in dropdown:
    arrow.click()

As I said, after this I need to find a way to get the html code after opening the dropdowns so that I can look for the price class. I have tried different things but I don't seem to get any working solution.

Thanks for your help.

what different things you have tried so far to extract the price? Show us your code snippet and html code. — theGuy
– theGuy, Commented Oct 4, 2018 at 15:58

Austen Novis · Accepted Answer · 2018-10-04 15:22:35Z

1

You can try using the Selenium WebDriverWait. WebDriverWait

WebDriverWait wait = new WebDriverWait(driver, 30);

WebElement element = wait.until(ExpectedConditions.presenceOfElementLocated(css));

answered Oct 4, 2018 at 15:22

Austen Novis

4442 gold badges12 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Moshe Slavin Over a year ago

You are correct!! The best way to automate with selenium is using WebDriverWait yet his question was about how to get all of the HTML code... Hence my answer...

Moshe Slavin · Accepted Answer · 2018-10-04 16:04:40Z

0

First, You should use WebDriverWait as Austen had pointed out.

For your question try this:

from selenium import webdriver


driver=webdriver.Chrome(executable_path=r"C:\webdrivers\chromedriver.exe")
driver.get(r.url)
dropdown=driver.find_elements_by_class_name("expandArrow")
for arrow in dropdown:
    arrow.click()
    html_source = driver.page_source
    print(html_source)

Hope this helps you!

answered Oct 4, 2018 at 16:04

Moshe Slavin

5,2445 gold badges27 silver badges40 bronze badges

2 Comments

Dani Garcia Martos Over a year ago

Thanks, this has worked, I have tried it using sleep instead of the WebDriverWait function because I don't really know how to use it yet (I'm currently looking into it) and I have gotten good results.

Moshe Slavin Over a year ago

🔼 I am glad to help! You really should use webdriverwait!

Collectives™ on Stack Overflow

Selenium: How to parse through a code after using selenium to click a dropdown

2 Answers 2

1 Comment

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related