Python Selenium use list of Urls

Question

I have a text list of urls named all_urls.txt. Each url in the text file is on a single line. I want to pass this list to selenium (python) to extract specific data. I can achieve this by using the url's one by one but this is not efficient. My code at present looks like this:-

profile = FirefoxProfile('/home/test/.mozilla/firefox/mfgrtrtr.Default3')
browser = webdriver.Firefox(firefox_profile=profile)
browser.maximize_window()
# get website
browser.get('https://www.some-website.com/')
# get current url
print browser.current_url
# get name & get phone number
name = browser.find_element_by_class_name("name")
print name.text
phone = browser.find_element_by_class_name("phone")
print phone.text

How can I pass the list to browser.get and extract name and phone from each url. Thanks in advance for your help, I am new to python but enjoying the challenge.

Do you know how to open a file and use a for loop? with open(yourfile) as f:for url in map(str.rstrip, f) ... — Padraic Cunningham
– Padraic Cunningham, Commented Mar 23, 2016 at 11:03

holdenweb · Accepted Answer · 2016-03-23 11:30:49Z

2

You probably need a for loop, which can iterate over a list. Your code should look something like this:

profile = FirefoxProfile('/home/test/.mozilla/firefox/mfgrtrtr.Default3')
browser = webdriver.Firefox(firefox_profile=profile)
browser.maximize_window()
with open("your_file_name") as in_file:
    for url in in_file:
        # get website
        browser.get(url.strip())
        # get current url
        print browser.current_url
        # get name & get phone number
        name = browser.find_element_by_class_name("name")
        print name.text
        phone = browser.find_element_by_class_name("phone")
        print phone.text

The .strip method call on the URL simply ensures that it has no leading or trailing whitespace - lines read in from a file normally include the trailing newline character.

edited Mar 23, 2016 at 11:30

answered Mar 23, 2016 at 11:00

holdenweb

37.8k7 gold badges62 silver badges80 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

D. O Sullivan · Accepted Answer · 2016-03-23 11:05:56Z

2

Open the file:

my_file = open("all_urls.txt", "r")

Iterate throught it and use the get function on each url:

for url in my_file:
    browser.get(url)
    print ...
    print ...

answered Mar 23, 2016 at 11:05

D. O Sullivan

1998 bronze badges

Collectives™ on Stack Overflow

Python Selenium use list of Urls

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related