Python web crawler no output

Question

I tried to create my first python web crawler (learned it from thenewboston). I dont get any error messages, but also no output.. Heres my code:

import requests
from bs4 import BeautifulSoup

def sportpoint_spider(max_pages):
    page = 1
    while page <= max_pages:
        url = 'http://www.sportpoint.lt/vyrams-1?page=' + str(page)
        source_code = requests.get(url)
        plain_text = source_code.text
        soup = BeautifulSoup(plain_text, "html.parser")
        for link in soup.findAll('a', {'atl '}):
            href = link.get('href')
            print(href)
        page += 1

sportpoint_spider(1)

Could you add print(plain_text) statement after plain_text = source_code.text and post results? — kvorobiev
– kvorobiev, Commented Oct 15, 2017 at 18:20
it printed all website text, classes and etc. (all text from inspect element) — pijasas
– pijasas, Commented Oct 15, 2017 at 18:24
Isn't that what its supposed to do? It runs and then exits. You need to save the output to a file. — Hunter
– Hunter, Commented Oct 15, 2017 at 18:29

kvorobiev · Accepted Answer · 2017-10-15 18:47:13Z

2

Your problems lays at this line

for link in soup.findAll('a', {'atl '}):

according to docs second argument attrs should be a dictionary with pairs like {'attr_name': 'attr_value'}. And {'atl '} is a set. Also, I think you mean 'alt', not 'atl'. Try to use

for link in soup.findAll('a'):

There aren't 'a' elements on page with attribute 'alt'.

answered Oct 15, 2017 at 18:47

kvorobiev

5,0704 gold badges32 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python web crawler no output

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related