Problems with Python Crawler

Question

I'm new to python and I am trying to code a crawler, my issue is that I cannot get the href links I am targeting to display in the console. Any help is appreciated, see below.

import requests
from bs4 import BeautifulSoup

def trade_spider(max_pages):
    page = 1
    while page <= max_pages:
        url = 'http://www.rent.ie/houses-to-let/renting_dublin/page_'+ str(page)
        source_code = requests.get(url)
        plain_text = source_code.text
        soup = BeautifulSoup(plain_text)
        special_divs = soup.findAll('div', {'class':'search_result_title_box'}) 
        for link in special_divs:
            gold = link.findAll('a')
            for link in gold:
                href = gold.get(link['href'])
                print(href)
        page += 1

trade_spider(3)

alecxe · Accepted Answer · 2017-03-01 22:31:16Z

I'm not sure where you've found that search_result_title_box class, I'd locate the links inside the elements with search_result class. The following code works for me:

import requests
from bs4 import BeautifulSoup


def trade_spider(max_pages):
    """Docstring here."""

    with requests.Session() as session:
        session.headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"}

        for page in range(1, max_pages):
            url = 'http://www.rent.ie/houses-to-let/renting_dublin/page_{page}'.format(page=page)
            response = session.get(url)

            soup = BeautifulSoup(response.content, "html.parser")
            for link in soup.select(".search_result h2 > a[href]"):
                print(link["href"])

if __name__ == '__main__':
    trade_spider(3)

Note the following improvements:

we are using the requests.Session() to improve on performance (underlying TCP connection is reused) and to configure common things like HTTP headers
we are using a real browser's User-Agent string
we are explicitly specifying html.parser as an underlying parser used by BeautifulSoup
.search_result h2 > a[href] is a CSS selector to match the desired links in the search result headers. The > means a direct parent-child relationship.

Collectives™ on Stack Overflow

Problems with Python Crawler

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related