Python Selenium. Parallel if loop

Question

import csv
import time
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from selenium import webdriver
from csv import reader
from selenium.webdriver.common.action_chains import ActionChains
from selenium.common.exceptions import NoSuchElementException


chrome_options = Options()
scroll = 5
chrome_options.add_experimental_option("useAutomationExtension", False)
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
header_added = False
header_added1 = False
url = "url"
driver = webdriver.Chrome(executable_path='C:/chromedriver.exe', options=chrome_options)
driver.maximize_window()
driver.get(url)
time.sleep(3)
search_city = input("Enter the city :")
res_n = input("Enter the Restaurant's name :")
search = driver.find_element_by_xpath('//input[@name="location"]').send_keys(search_city)
time.sleep(2)
driver.find_element_by_xpath('//*[@id="root"]/div[1]/div[1]/div/div[1]/div[1]/div/div[2]/div/div[3]/div[1]/span[2]').click()
time.sleep(3)
driver.find_element_by_xpath('/html/body/div[1]/div[1]/header/div/div/ul/li[5]/div/a/span[1]').click()
time.sleep(1)
search_res = driver.find_element_by_class_name('_2BJMh').send_keys(res_n.lower())
time.sleep(5)
driver.find_element_by_class_name('_2BJMh').send_keys(Keys.RETURN)
time.sleep(5)

try:
    driver.find_element_by_class_name('_3FR5S').click()
    time.sleep(5)
except:
    print("restaurant not open")
    driver.quit()

html = driver.find_element_by_tag_name('html')



def get_items():
    global header_added
    global item_dvs
    cats = driver.find_elements_by_class_name('D_TFT')
    cats[1].click()
    time.sleep(3)
    item_dvs = driver.find_elements_by_class_name('_2wg_t')

    for div in item_dvs:
        name = div.find_element_by_class_name('styles_itemNameText__3bcKX')
        print(name.text)
        price = div.find_element_by_class_name('rupee')
        print(price.text)
        if div.find_elements_by_class_name('styles_itemDesc__MTsVd'):
            desc = div.find_element_by_class_name('styles_itemDesc__MTsVd').text
        else:
            desc = None
        if div.find_element_by_css_selector('div._1C1Fl._23qjy'):
            element = div.find_element_by_css_selector('div._1C1Fl._23qjy')
            print("found")
            driver.execute_script("arguments[0].scrollIntoView();", element)
            add = div.find_element_by_css_selector('._1RPOp')
            driver.execute_script("arguments[0].click();", add)
            time.sleep(1)
            add_ons = driver.find_element_by_class_name('_3UzO2').text
            print(add_ons)
            driver.find_element_by_css_selector('#modal-placeholder > div:nth-child(3) > div > div._1Kr-y._3EeZR > div > div._1EZLh > div > button').click()

        else:
            add_ons = None
        dict1 = {'Item Name': name.text, "Price": price.text, "Add Ons :": add_ons, "Description": desc}
        with open(f'{search_city}_{res_n}.csv', 'a+', encoding='utf-8-sig') as f:
            w = csv.DictWriter(f, dict1.keys())
            if not header_added:
                w.writeheader()
                header_added = True
            w.writerow(dict1)


get_items()

The is_cust loop keeps running over and over again opening the same element, while the rest of the code moves on to the next divs. What is wrong here?

I never use xPath, but they are bidirectional tutorialspoint.com/…... which might be in cause here. If you were using cssSelector, your div.find_element_by_css_selector would only retrieve the Childs of the parent div — Nic Laforge
– Nic Laforge, Commented Dec 27, 2020 at 18:50
Also your if/else statement is wrong. Your is_cust will either return an element or raise an exception. It can never be false — Nic Laforge
– Nic Laforge, Commented Dec 27, 2020 at 18:52
You are using a parent element to find a nested element (div.find...), but xPath won't care much about it from what I read. I provided an answer, but without html code it is hard to give you an accurate explanation/solution. One thing a noticed is that you are mixing (driver and div) to search your elements. Why? — Nic Laforge
– Nic Laforge, Commented Dec 27, 2020 at 19:00

Nic Laforge · Accepted Answer · 2020-12-27 19:52:57Z

1

xPath are bidirectional and is probably the cause here.

Try this code using cssSelector:

for div in item_dvs:
    #Do Something

    try:   
        is_cust = div.find_element_by_css_selector('._1C1Fl._23qjy')
        print("found")
    except NoSuchElementException:
        continue

    driver.execute_script("arguments[0].scrollIntoView();", is_cust)
    add = div.find_element_by_css_selector('._1RPOp')
    driver.execute_script("arguments[0].click();", add)
    time.sleep(1)
    # Not sure why for this one you had driver instead of div. Suspect div should be 
    add_ons = div.find_element_by_class_name('_26cJ9').text
    div.find_element_by_css_selector('#modal-placeholder > div:nth-child(3) >   div > div._1Kr-y._3EeZR > div > div._1EZLh > div > button').click()

UPDATE From your updated code, you are using lot of hardcoded sleep. I will suggest to use the WebDriverWait with expected_conditions.

More info here: Wait from Selenium

Imports needed:

from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

Code to be added post driver creation:

wait_time = 5 wait = WebDriverWait(driver, wait_time)

Instead of using sleep like this:

time.sleep(5)
driver.find_element_by_class_name('_2BJMh').send_keys(Keys.RETURN)
time.sleep(5)

Use:

wait.until(EC.presence_of_element_located((By.CLASS_NAME, '_2BJMh'))).send_keys(res_n.lower())

Don't gather the element twice.. use find_elements_by* then validate the length:

descs = wait.until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'styles_itemDesc__MTsVd')))
if len(descs) > 0:
    desc = descs[0].text
else:
    desc = None

edited Dec 27, 2020 at 19:52

answered Dec 27, 2020 at 18:58

Nic Laforge

1,8861 gold badge11 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

Abhishek Rai Over a year ago

You see the click. It is a different window. That is why the driver. Also, I tried with the css. Same result. I will re run it just to make sure.

Abhishek Rai Over a year ago

It worked !..Alas only for a few divs. Looks like you have your finger on the problem..Not sure why it isn't working for the rest...let me check.

Nic Laforge Over a year ago

Nice.. getting there! Can you provide more info when it does not work. If you open a new window/pop-up you might need to introduce WebDriverWait to allow the element to be loaded in the DOM

Nic Laforge Over a year ago

Are you able to share the html/site. And if so, provide how you retrieve item_dvs

Abhishek Rai Over a year ago

No, it's like you mentioned. When it doesn't find the element it doesn't go to else. Throws an error.

|

Collectives™ on Stack Overflow

Python Selenium. Parallel if loop

1 Answer 1

10 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

10 Comments

Your Answer

Sign up or log in

Post as a guest

Related