Manipulating web pages with python

Question

My aim is to be able to automatically manipulate a web page with a script. Filling in information and selecting the correct drop down box. With minimal user input.

So my example here I'm using the national rail website.

import win32com.client
from time import sleep

ie = win32com.client.Dispatch("InternetExplorer.Application")
ie.Visible = 1
ie.navigate("http://www.nationalrail.co.uk/")

while ie.ReadyState != 4: # Wait for browser to finish loading
    sleep(1)
print("Webpage Loaded")

page = ie.Document

links = page.links

If I wanted to change the box When "Leaving" to arriving, fill in the From "Station / Postcode" and hit Go. How would I go about doing this?

Also is win32com the best method for manipulating webpages like this?

I'll answer your second question first. The standard for browser simulation is the selenium module for Python. If you use this option, you can directly run some sort of JavaScript on the page to manipulate the page. The beautiful soup or bs4 module is used for scraping the page for links. — rassa45
– rassa45, Commented Aug 15, 2016 at 16:48

n1c9 · Accepted Answer · 2016-08-15 16:54:58Z

2

While I'm sure every Python user can appreciate you trying to do this in the most difficult way possible, why not make things easier for yourself and use the library Selenium?

Here's your code & what you're trying to do, in Selenium:

from selenium import webdriver
driver = webdriver.Firefox() # Initialize the webdriver session
driver.get('http://www.nationalrail.co.uk/') # replaces "ie.navigate"
driver.find_element_by_id('sltArr').find_elements_by_tag_name('option')[1].click() # Selects the "Arrive" option

See? Much better looking! That last line selects the "Leaving" form, finds the option tags within it, and selects the Arrive option. With this bit of code, you should be able to figure out the rest of what you want to do with this site as well.

answered Aug 15, 2016 at 16:54

n1c9

2,6874 gold badges35 silver badges55 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Manipulating web pages with python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related