issue with scraping using beautiful soup and selenium using python

Question

Im trying to practice using selenium / BS4 and ive ran into an issue.

basically im trying to get a url from:

`'class':'td_text_highlight_marker_green td_text_highlight_marker'`

but whenever i run the script the returned number of links is always different.

everytime it runs it should return 18. I even tried to click the link to follow it using selenium and for the life of me i cannot figure it out.

please help :)

Edit code to your post please.

0m3r
– 0m3r

2020-05-28 00:10:38 +00:00
Commented May 28, 2020 at 0:10 — 0m3r
– 0m3r, Commented May 28, 2020 at 0:10

0buz · Accepted Answer · 2020-05-28 11:26:54Z

1

The webpage DOM does not load fully in time for your search. So to ensure the span is actually there before you start looking for it with BeautifulSoup, add a WebDriverWait.

for coupon_url in coupon_url_list:
    driver.get(coupon_url)
    WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//span[@class='td_text_highlight_marker_green td_text_highlight_marker']")))
    content = driver.page_source
    # rest of your code here

answered May 28, 2020 at 11:26

0buz

3,5352 gold badges12 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

issue with scraping using beautiful soup and selenium using python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related