How to skip errors in web scraping with python?

Question

I am new in python and would like to learn web scraping with python. My first project are the yellow pages in Germany.

When executing my code, I am getting following IndexError after scraping 12 pages:

('Traceback (most recent call last): File "C:/Users/Zorro/PycharmProjects/scraping/venv/Lib/site-packages/pip-19.0.3-py3.6.egg/pip/_vendor/pytoml/test.py", line 25, in city = city_container[0].text.strip() IndexError: list index out of range

Process finished with exit code 1')

I would like to know how I can skip this error, so that python does not stop scraping.

I tried to use try and except blocks, but did not succeed.

from bs4 import BeautifulSoup as soup
import requests


page_title = "/Seite-"
page_number = 1

for i in range(25):

my_url = "https://www.gelbeseiten.de/Branchen/Italienisches%20Restaurant/Berlin"

page_html = requests.get(my_url + page_title + str(page_number))
page_soup = soup(page_html.text, "html.parser")

containers = page_soup.findAll("div", {"class": "table"})

for container in containers:
    name_container = container.findAll("div", {"class": "h2"})
    name = name_container[0].text.strip()

    street_container = container.findAll("span", {"itemprop": "streetAddress"})
    street = street_container[0].text.strip()

    city_container = container.findAll("span", {"itemprop": "addressLocality"})
    city = city_container[0].text.strip()

    plz_container = container.findAll("span", {"itemprop": "postalCode"})
    plz_name = plz_container[0].text.strip()

    tele_container = container.findAll("li", {"class": "phone"})
    tele = tele_container[0].text.strip()

    print(name, "\n" + street, "\n" + plz_name + " " + city, "\n" + tele)
    print()

page_number += 1

Please read the docs about exception handling.

ForceBru
– ForceBru

2019-08-13 19:28:10 +00:00
Commented Aug 13, 2019 at 19:28 — ForceBru
– ForceBru, Commented Aug 13, 2019 at 19:28

seulberg1 · Accepted Answer · 2019-08-13 19:39:19Z

Ok, the formatting seems to have suffered a little upon posting the code. Two things:

1) When webscraping it is usually advisable to add some downtime between consecutive scrapes to not get thrown off the server and not block too many resources. I added time.sleep(5) between every page request to wait 5 seconds before loading another page.

2) For me, try except worked just fine, if you add pass to the exception part. Of course, you can become more sophisticated in treating exceptions.

from bs4 import BeautifulSoup as soup
import requests
import time


page_title = "/Seite-"
page_number = 1

for i in range(25):
    print(page_number)
    time.sleep(5)
    my_url = "https://www.gelbeseiten.de/Branchen/Italienisches%20Restaurant/Berlin"

    page_html = requests.get(my_url + page_title + str(page_number))
    page_soup = soup(page_html.text, "html.parser")

    containers = page_soup.findAll("div", {"class": "table"})

    for container in containers:

        try:
            name_container = container.findAll("div", {"class": "h2"})
            name = name_container[0].text.strip()

            street_container = container.findAll("span", {"itemprop": "streetAddress"})
            street = street_container[0].text.strip()

            city_container = container.findAll("span", {"itemprop": "addressLocality"})
            city = city_container[0].text.strip()

            plz_container = container.findAll("span", {"itemprop": "postalCode"})
            plz_name = plz_container[0].text.strip()

            tele_container = container.findAll("li", {"class": "phone"})
            tele = tele_container[0].text.strip()

            print(name, "\n" + street, "\n" + plz_name + " " + city, "\n" + tele)
            print()

        except:
            pass

    page_number += 1

Collectives™ on Stack Overflow

How to skip errors in web scraping with python?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related