Selenium - scraping dynamically generated data Python

Question

I am trying to get data from a dynamically generated web page. From my searches I found that Selenium is probably the best option to go with but I am running into some issues. The web page that I want to get data from is this one and my test search data is "10403782"

So far I have the following source code that is able to locate the search bar and search but as you can see the result back is multiple items, and I am trying to locate the one that doesn't have the small house to the left greyed out.

# Initial connection and search
driver.get("http://firmaopslag.dk")
element = driver.find_element_by_id("firmanavn")
element.send_keys("10403782")
element.send_keys(Keys.RETURN)

# On search result page, find the result with the house
searchResults = driver.find_element_by_id("searchresult")

I think that one way to locate the blue house is by looking at the color value, loop through all result items and find the one that doesn't have the house color that matches the grey one. However, whenever I make a search as in the case above searchResults is always empty. I tried searching by class name, id, tags .. nothing appears to be able to locate the results. Essentially as I mentioned I want to locate the result with the blue house and click on it.

EDIT: I think my biggest issue is that once the search is done I need to be looking at a different web page or a different element that what I've had so far from the initial page

Also for the final part, once I am on the correct page, I think beautifulsoup is the best way to get the data I am interested in, isn't it?

alecxe · Accepted Answer · 2016-01-19 00:19:47Z

1

You can check the color inside the style attribute:

# Initial connection and search
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox()
driver.get("http://firmaopslag.dk")
element = driver.find_element_by_id("firmanavn")
element.send_keys("10403782")
element.send_keys(Keys.RETURN)

# wait for search results to appear
wait = WebDriverWait(driver, 10)
searchResults = wait.until(EC.presence_of_element_located((By.ID, "searchresult")))

for blue_house_result in searchResults.find_elements_by_xpath(".//button[.//span[contains(@class, 'glyphicon-home') and contains(@style, 'color: #002954;')]]"):
    label = blue_house_result.find_element_by_tag_name("h4")
    print(label.text)

Note that I've also added a wait for the search results to appear after the search is performed.

Also for the final part, once I am on the correct page, I think beautifulsoup is the best way to get the data I am interested in, isn't it?

You can use BeautifulSoup to further parsing HTML from the driver.page_source, but it is not necessarily needed since you can locate elements with selenium.

edited Jan 19, 2016 at 0:19

answered Jan 18, 2016 at 23:32

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Slavi Over a year ago

This is amazing, thank you! And if I wanted to click the blue_house_result instead of printing the text, blue_house_result.click() should do it right?

alecxe Over a year ago

@Slavi yup, it should. Note that there can also be pagination depending on the search query.

Slavi Over a year ago

I see, by the way, there's only 1 blue house possible so i changed to find_element_by_xpath, however the click() doesn't work I think it's now pointing to the elements within the button not the button it self, is that correct? Perhaps a workaround is to loop through all buttons and when i find the one with the blue house to click that or is there easier way to refer to the button that blue_house_result data came from?

Slavi Over a year ago

Interesting ... The only difference with your solution is that I am using Chrome but tried on 3 separate physical machines, 1 linux and 2 windows and neither seemed to work ... I'll try with firefox, cheers again!

Slavi Over a year ago

update .. got it working with firefox on Linux, thank you! Kind of curious why it wouldn't work in chrome .. kept scrolling down the page but didn't click on anything at all just returned errors, oh well :)

Collectives™ on Stack Overflow

Selenium - scraping dynamically generated data Python

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related