web scraping selenium find elements by class name

Question

I try to extract the all the products data from this page:

https://www.shufersal.co.il/online/he/קטגוריות/סופרמרקט/חטיפים%2C-מתוקים-ודגני-בוקר/c/A25

I want to get all the li tag with the name class miglog-prod miglog-sellingmethod-by_unit

like this:

<li class="miglog-prod miglog-sellingmethod-by_unit" data-product-name="צדפים דגנים בטעם שוקולד" data-entry-number=""
data-product-box
data-product-replace=""
data-product-price="12.9"
data-product-purchasable="false"
data-food="true"
data-selling-method="BY_UNIT"
data-product-code="P_7296073442226">

I tried:

shufersal = "https://www.shufersal.co.il/online/he/קטגוריות/סופרמרקט/חטיפים%2C-מתוקים-ודגני-בוקר/c/A25"

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
    
driver = webdriver.Chrome(ChromeDriverManager().install())
    
import time
driver.get(shufersal)
for i in range(0,5): # here you will need to tune to see exactly how many scrolls you need
    driver.execute_script('window.scrollBy(0, 400)')
    time.sleep(1)
products = driver.find_elements_by_class_name("miglog-prod miglog-sellingmethod-by_unit")
    
print(products)

but I don't find any product:

[]

Shod · Accepted Answer · 2021-12-09 13:40:11Z

1

A couple of things you can do:

from selenium.webdriver.common.by import By

products = driver.find_element_by_class_name(
    By.CSS_SELECTOR,
    ".miglog-prod.miglog-sellingmethod-by_unit"
)

or,

products = driver.find_elements_by_css_selector(
    "li.miglog-prod.miglog-sellingmethod-by_unit"
)

Note that in the first method, I am passing just the class names, but in second one I am passing even the element (li).

(Edit: Ok, I checked on REPL, I get elements even if I don't pass li so I am not sure about the behaviour here. Btw, I tested this for Firefox.)

From what I read, you dont get products is because find_elements_by_class_name needs only one class name and you have passed two..

edited Dec 9, 2021 at 13:40

answered Dec 9, 2021 at 13:33

Shod

9553 gold badges12 silver badges39 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

web scraping selenium find elements by class name

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related