TypeError issue with WebDriverWait (Python/Selenium)

Question

I am having trouble using explicit wait with Selenium. I keep getting the following error.

File "C:/Users/username/path/craigslist_scrape.py", line 108, in <module>
    WebDriverWait(driver,5).until(EC.text_to_be_present_in_element_value(By.XPATH('//p[@class="reply-email-address"]/a[@class="mail app"]')))

TypeError: 'str' object is not callable

My code is as follows. Basically, I am trying to scrape the email address. However, most of the time, the email address is not visible unless I click 'reply'. What I am trying to do is to get Selenium to click Reply and wait until the email address is visible and then scrape. I have tried using a couple of different methods, but kept getting the same error. Not sure what exactly happened. Hope somebody can help.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.select import Select
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import TimeoutException

driver=webdriver.Chrome()
driver.get('https://nh.craigslist.org/search/reb')

link_list=[]

link=driver.find_elements_by_xpath('//li[@class="result-row"]/a[@href]')
for elem in link:
    print(elem.get_attribute('href'))
    link_list.append(elem.get_attribute('href'))
for link in link_list:
    driver.get(link)
    driver.find_element_by_xpath('//div[@class="actions-combo"]/button[@role="button"]').click()
    WebDriverWait(driver,5).until(EC.text_to_be_present_in_element_value(By.XPATH('//p[@class="reply-email-address"]/a[@class="mail app"]')))
    email=driver.find_element_by_xpath('//p[@class="reply-email-address"]/a').text
    print(email)

I think you should be more specific, where this error appeared ... because I would have to run the code to find out, what could went wrong ... Please provide a full error log :) — StyleZ
– StyleZ, Commented Apr 20, 2020 at 0:39
It's with the WebDriverWait function. Can't get it to work. And how do you get the full error log? Sorry a bit new to coding. — Fatchoco
– Fatchoco, Commented Apr 20, 2020 at 1:05
no at all :) When it gives an error, just copy paste it, I am sure that its not just the 1 line error :D — StyleZ
– StyleZ, Commented Apr 20, 2020 at 1:06
WebDriverWait(driver,5).until(EC.text_to_be_present_in_element_value(By.XPATH('//p[@class="reply-email-address"]/a[@class="mail app"]'))) TypeError: 'str' object is not callable — Fatchoco
– Fatchoco, Commented Apr 20, 2020 at 1:12
This is where it throws the error. I have tried a variety of ECs, but can't get it to work. Basically I am trying to make sure that the element is present before I can scrape it. — Fatchoco
– Fatchoco, Commented Apr 20, 2020 at 1:13

StyleZ · Accepted Answer · 2020-04-20 01:26:14Z

2

Okay I have realised what you want to do ...

The problem is that you are calling the wrong function with wrong parameters. Here is the correct way to call it :)

WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, '/html/body/section/section/header/div[2]/div/div[1]/aside')))

The main problem is that you are telling the function that which element it should be looking for using which function ...

Here is SO answer that answers it better than I do: Selenium - wait until element is present, visible and interactable

answered Apr 20, 2020 at 1:26

StyleZ

1,2733 gold badges15 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Fatchoco Over a year ago

This works. Thank you! Just wondering if scraping not visible elements with BeautifulSoup is faster than with Selenium?

StyleZ Over a year ago

I am not sure, but if you care about speed, I think you chose a wrong language for the job :D ... also, please accept the answer

Collectives™ on Stack Overflow

TypeError issue with WebDriverWait (Python/Selenium)

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related