1

I'm trying to scrape some data from one flight-searching web page. It is probably generated with Javascript. I've tried many approaches but nothing works so I've decided to try selenium. 

from selenium import webdriver

driver = webdriver.Firefox()
driver.get('https://www.pelikan.sk/sk/flights/list?dfc=CVIE%20BUD%20BTS&dtc=CMAD&rfc=CMAD&rtc=CVIE%20BUD%20BTS&dd=2015-07-09&rd=2015-07-14&px=1000&ns=0&prc=&rng=1&rbd=0&ct=0')
print driver.page_source

I though that it return final javascript-generated html code but I can't find there strings which are on this page when open it in browser.

Where could be the problem? What should I do to get those flights?

EDIT: I forgot to mention that the page is continualy loading new flights. So when you open it in a browser it shows some flights but it still loading other flights.

Improve this question

1 Answer 1

Reset to default
1

The page has quite a dynamic nature and you need to wait for the page to load. Choose something that would indicate that a page and search results were loaded. For instance, wait until the loading image (with a pelican) becomes invisible:

from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC


driver = webdriver.Firefox()
driver.get("https://www.pelikan.sk/sk/flights/list?dfc=CVIE%20BUD%20BTS&dtc=CMAD&rfc=CMAD&rtc=CVIE%20BUD%20BTS&dd=2015-07-09&rd=2015-07-14&px=1000&ns=0&prc=&rng=1&rbd=0&ct=0")

wait = WebDriverWait(driver, 60)
wait.until(EC.invisibility_of_element_located((By.XPATH, '//img[contains(@src, "loading")]')))
wait.until(EC.invisibility_of_element_located((By.XPATH, u'//div[. = "Poprosíme o trpezlivosť, hľadáme pre Vás ešte viac letov"]/preceding-sibling::img')))

print(driver.page_source)

Here we are waiting for two pelicans to fly away disappear: a bigger one and a smaller one.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.