How to get selenium web driver on python to find elements on css selectors of a following page?

Question

I am trying to get selenium to web scrape the first paragraph of wiki pages using CSS selectors.

When I run this code, it seems to only select ones from the original web page

https://en.wikipedia.org

and not what I am searching for, in this case 'cats'.

Any help with this would be awesome!


browser = webdriver.Firefox(executable_path='D:\Import Files that I also want backed up\Jupyter Notebooks\Python Projects\Selenium\driverss\geckodriver.exe')
browser.get('https://en.wikipedia.org')

search_elem = browser.find_element_by_css_selector('#searchInput')

search_elem.send_keys('cats')
search_elem.submit()


results_elem = browser.find_element_by_css_selector('p')

print(results_elem.text)

output:

Adventure Time is an American fantasy animated television series created .....

I want to print the first paragraph of the 'cat' page. But when using the css selectors I am still only scraping off the first 'wikipedia.com' page. Even though I am on the 'cat' page. Essentially I want to be able to scrape from a web page after searching a topic using selenium. — John Cook
– John Cook, Commented Apr 5, 2020 at 21:03

KunduK · Accepted Answer · 2020-04-05 20:34:47Z

1

To get the first paragraph text from wiki page.Induce WebDriverWait() and visibility_of_element_located() and following css selector.

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

browser = webdriver.Firefox(executable_path='D:\Import Files that I also want backed up\Jupyter Notebooks\Python Projects\Selenium\driverss\geckodriver.exe')
browser.get('https://en.wikipedia.org')
search_elem = browser.find_element_by_css_selector('#searchInput')
search_elem.send_keys('cats')
search_elem.submit()
results_elem=WebDriverWait(browser,10).until(EC.visibility_of_element_located((By.CSS_SELECTOR,"div.mw-parser-output p:nth-of-type(3)")))
print(results_elem.text)

answered Apr 5, 2020 at 20:34

KunduK

33.4k5 gold badges19 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

John Cook Over a year ago

see this works if I eliminate the code using the search bar. But when I get to the cat page by searching for cats it takes that css selector from the first page visited 'en.wikipedia.org'

KunduK Over a year ago

If your page is taking more time to load.then provide some time.sleep(5) after submit the page.let me know how this goes?

Collectives™ on Stack Overflow

How to get selenium web driver on python to find elements on css selectors of a following page?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related