Scrape multiple pages with loops in Python

Question

I successfully scraped the first page of the website, but when I tried to scrape mutiples pages, it worked but the result is totally wrong.

Code:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
for num in range(1,15):
    res = requests.get('http://www.abcde.com/Part?Page={num}&s=9&type=%8172653').text
    soup = BeautifulSoup(res,"lxml")
    for item in soup.select(".article-title"):
        print(urljoin('http://www.abcde.com',item['href']))

It only changed one number in every page's url, for example,

http://www.abcde.com/Part?Page=1&s=9&type=%8172653
http://www.abcde.com/Part?Page=2&s=9&type=%8172653

I got total 14 pages of this.

My code worked, but it just repeatedly print out the first page's url for 14 times. The result I expected was to print out all different urls from different pages using loops.

You're not actually formatting the string to replace the number into it... So you either need to prefix the string with f if you're using 3.6+ or otherwise .format(num=num) the string to put the page number in... — Jon Clements
– Jon Clements, Commented Oct 12, 2017 at 10:07

allo · Accepted Answer · 2017-10-12 11:51:49Z

3

As Jon Clements pointed, format url as below :

res = requests.get('http://www.abcde.com/Part?Page={}&s=9&type=%8172653'.format(num)).text

You can find more about python format strings at pyformat.info.

edited Oct 12, 2017 at 11:51

allo

4,3088 gold badges45 silver badges76 bronze badges

answered Oct 12, 2017 at 10:09

Dinesh Pundkar

4,1962 gold badges26 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Makiyo Over a year ago

Hi! Thanks for the info. I tried, but it said AttributeError: 'Response' object has no attribute 'format'

Dinesh Pundkar Over a year ago

Sorry my bad. Missed one round bracket at last. Updated the code

Collectives™ on Stack Overflow

Scrape multiple pages with loops in Python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related