Get URL in Selenium Python

Question

I am very new to python and I am looking to scrape following website:Link

I think that Selenium might be the right tool and I started to write following code:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

path='http://planning.hackney.gov.uk/Northgate/PlanningExplorer/generalsearch.aspx'

browser = webdriver.Firefox()
browser.get(path)

elem = browser.find_element_by_id('txtPostCode')
elem.clear()
elem.send_keys("E9 7JP")
elem.send_keys(Keys.RETURN)

print (browser.current_url)

So far so good, it works. However, the return value of browser.current_url is not quite what is displayed in the url bar of my browser. I mean the the return value of the script is:

//planning.hackney.gov.uk/Northgate/PlanningExplorer/generalsearch.aspx

however the url in the browser is showing me this one here:

//planning.hackney.gov.uk/Northgate/PlanningExplorer/Generic/StdResults.aspx?PT=Planning%20Applications%20On-Line&SC=Postcode%20is%20E9%207JP&FT=Planning%20Application%20Search%20Results&XMLSIDE=/Northgate/PlanningExplorer/SiteFiles/Skins/Hackney/Menus/PL.xml&XSLTemplate=/Northgate/PlanningExplorer/SiteFiles/Skins/Hackney/xslt/PL/PLResults.xslt&PS=10&XMLLoc=/Northgate/PlanningExplorer/Generic/XMLtemp/j5jzxiwxklgslnam4qffypw5/052dd052-3993-4f10-83aa-dd0c6c326676.xml

Now I wonder how to get hold of this adress?!

Thanks a lot!

Could you add your Python version, python-selenium version, firefoxdriver version and firefox version to the post? I could not reproduce your issue using python3, python-selenium 2.53, firefoxdriver 2.53 and Firefox 45.9.0. Running your scripts gives me longer URL, just like you expected. — Mirek Długosz
– Mirek Długosz, Commented Apr 30, 2017 at 17:30
thanks for your answer. my python version is 3.6.1, selenium is 3.4.0, Firefox is 53, i have no idea how to figure out the version of the Firefox driver, but it's the newest one. l installed selenium just a few days ago.. — flow.v
– flow.v, Commented May 1, 2017 at 20:30

demouser123 · Accepted Answer · 2017-04-30 17:29:38Z

3

Did you made any other request in between checking your script returned URL and the URL shown by the browser. The request URL sent post the Keys.RETURN adds a session identifier with the URL, which might be the reason why you are getting different URL.

I have this script

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
chromepath='chrome_driver_path' //change this to your chromedriver path
driver = webdriver.Chrome(chromepath)

driver.get('http://planning.hackney.gov.uk/Northgate/PlanningExplorer/generalsearch.aspx')

print(driver.current_url)

elem = driver.find_element_by_id('txtPostCode')
elem.clear()
elem.send_keys("E9 7JP")
elem.send_keys(Keys.RETURN)

print (driver.current_url)

driver.quit()

Keypress code has been copied from your code itself. I get an identical URL from both the browser and the script

Script gives me this URL - Link Browser gives me this same URL - Copied Manually

answered Apr 30, 2017 at 17:29

demouser123

4,27212 gold badges54 silver badges88 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

flow.v Over a year ago

perfect, thanks a lot! it works! the same code give me different results depending on the driver i am using... meaning that the Firefox driver gives me the short address, the chrome driver the long one and the one i need...

Collectives™ on Stack Overflow

Get URL in Selenium Python

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related