webscraping code not working-python

Question

So i'm brand new to python wrote abit of code for webscraping average prices for houses throughout the UK but it hardly every works for lists > 5. This is normally due to the website because when i hit it again it works...but only sometimes.

the price variable used to be one long statement but it kept messing up, so i split it into a for statement and this keeps popping up:

Traceback (most recent call last):
  File "airbnb.py", line 18, in <module>
    town = item [i]
  File "//anaconda/lib/python3.5/site-packages/bs4/element.py", line 997, in __getitem__
    return self.attrs[key]
KeyError: 1

this is my entire code:

df = pd.read_csv('towns.csv')
item=df.test
name=[]
l1=[]
date=[]

for i in range(len(item)):
    town = item [i]
    url='https://www.airbnb.co.uk/s/?page=1&source=filters&airbnb_plus_only=false&room_types%5B%5D=Entire%20home%2Fapt&ss_id=6eaibfgb&allow_override%5B%5D=&s_tag=QmOEOer6'
    index=url.find('?page=')
    url=url[:index] + str(town) + url[index:]
    r=requests.get(url)
    soup=BeautifulSoup(r.content,'lxml')
    price=(soup.find('div',{"class":"avg-price"}))
    for item in price:
        j=(soup.find('span',{"class":"price"}).text)
    j=j [1:][:3]
    l1.append(j)
    name.append(town)
    time.sleep(1)


    stamp=time.strftime('%x')

    date.append(stamp)

    print(town)
    print(l1)
    print(date)

I suppose the problem could be the space inserted between item and [i], Try "town = item[i]" maybe. — Ukimiku
– Ukimiku, Commented Oct 27, 2016 at 10:21

IgnacioF · Accepted Answer · 2016-10-27 10:23:09Z

1

I think the error is in j=j [1:][:3] I recommend that before doing that you check the length of the j variable because it may be too short.

answered Oct 27, 2016 at 10:23

IgnacioF

513 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

reuben Over a year ago

Thanks, ill give it a change around

brianpck Over a year ago

Nope, Python slicing handles error checking of this kind: [][1:][:3] == []

Collectives™ on Stack Overflow

webscraping code not working-python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related