python selenium: how to navigate to certain page tab in the web page

Question

I was doing web scraping for a website with multiple pages in one web page. But when I click page 2, the url showed http://www.worldhospitaldirectory.com/Germany/hospitals#page-2.

And I put this url as next navigation location. And it goes directly to http://www.worldhospitaldirectory.com/Germany/hospitals#page-1, which is the default page.

I don't how to navigate to these sub pages. Any suggestions or code?

my code now:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Firefox()
driver.get('http://www.worldhospitaldirectory.com/Germany/hospitals')
url = []
pagenbr = 1

while pagenbr <= 43:
   current = driver.current_url
   driver.get(current)
   lks = driver.find_elements_by_xpath('//*[@href]')
   for ii in lks:
       link = ii.get_attribute('href')
       if '/info' in link:
           url.extend(link)
           print (link)
   print('page ' + str(pagenbr) + ' is done.')
   elm = driver.find_element_by_link_text('Next')
   driver.implicitly_wait(10)
   elm.click()
   pagenbr += 1

Can you provide the code that you're using?

tblznbits
– tblznbits

2017-02-06 14:54:16 +00:00
Commented Feb 6, 2017 at 14:54 — tblznbits
– tblznbits, Commented Feb 6, 2017 at 14:54
Sure. I will update my code there.@brittenb

Peter Cui
– Peter Cui

2017-02-06 15:03:42 +00:00
Commented Feb 6, 2017 at 15:03 — Peter Cui
– Peter Cui, Commented Feb 6, 2017 at 15:03

Andersson · Accepted Answer · 2017-02-06 15:01:45Z

2

Try just to click appropriate button on pagination as

driver.find_element_by_link_text('Next') # to get next page

or

driver.find_element_by_link_text('2') # to get second page

answered Feb 6, 2017 at 15:01

Andersson

52.8k18 gold badges83 silver badges132 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

Peter Cui Over a year ago

I updated my code. It worked on iterate to a new page. But after I iterate to new one. My code cannot pull the links as it did first time. Please give me some advices

Andersson Over a year ago

What you expect this line url.extend(link) to do? Do you mean url.append(link) ?

Peter Cui Over a year ago

Yes. I forgot to change it to append.

Peter Cui Over a year ago

Do you know what is wrong with my code. After it get the links from the first page, it cannot get other links from page 2, 3 and etc.

Andersson Over a year ago

What is output of your code? do you get any exceptions, something about stale element?

|

Wonka · Accepted Answer · 2017-02-06 15:11:12Z

1

Get element button

button_next = driver.find_element_by_xpath('//a[@class='page-link next'])
button_next.click()

I let the algorithm to iterate all pages for you

answered Feb 6, 2017 at 15:11

Wonka

1,9012 gold badges15 silver badges23 bronze badges

4 Comments

Peter Cui Over a year ago

Thx, but after I iterate to each new page, i cannot make my loop to pull links from new page. Would you like to take a look? I will update my code now.

Wonka Over a year ago

Probably you will have to sleep when click, because script execution is faster than the web load

Peter Cui Over a year ago

yeah, but i put waiting time there to wait for fully loading.

Wonka Over a year ago

No, driver.implicitly_wait(10) is NOT like sleep, is the MAX time to wait functions like find_element to find an element in the web.

Community · Accepted Answer · 2020-06-20 09:12:55Z

0

This worked for me

while pagenbr <= 3:
    current = driver.current_url
    print current
    driver.get(current)
    lks = driver.find_elements_by_xpath('//*[@href]')
    for ii in lks:
        link = ii.get_attribute('href')
        if '/info' in link:
            url.extend(link)
            print (link)
    print('page ' + str(pagenbr) + ' is done.')
    elm = driver.find_element_by_link_text('Next')
    driver.implicitly_wait(10)
 
    elm.click()
    driver.implicitly_wait(10)
    lks = driver.find_elements_by_xpath('//*[@href]')
    for ii in lks:
        link = ii.get_attribute('href')
        if '/info' in link:
            url.extend(link)
            print (link)

 
    pagenbr += 1

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Jul 21, 2019 at 2:46

sanjay

11 bronze badge

Collectives™ on Stack Overflow

python selenium: how to navigate to certain page tab in the web page

3 Answers 3

9 Comments

4 Comments

This worked for me

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

9 Comments

4 Comments

This worked for me

Comments

Your Answer

Sign up or log in

Post as a guest

Related