error in using BeautifulSoup python

Question

I'm newbie to HTML parsers. I'm actually trying to parse the source code of the webpage with url (http://www.quora.com/How-many-internships-are-necessary-for-a-B-Tech-student). I'm trying to get the answer_count.

I tried it in the following way:

import urllib2
from bs4 import BeautifulSoup

q = urllib2.urlopen(url)
soup = BeautifulSoup(q)
divs = soup.find_all('div',class_='answer_count')

But I get the list 'divs' as empty. Why is it so? Where am I wrong? How do I implement it to get the result as '2 Answers'?

There is a answer_count class in the source code! Here's a small patch: <div class="answer_count">2 Answers<span id="ld_bdnqjl_196692"></span></div> — Wasim Thabraze
– Wasim Thabraze, Commented Jul 23, 2014 at 12:04
I agree with MA1, there is no answer_count in the source that I load. I think you are looking at the source from being logged in as opposed to what urllib2 grabs. Try looking at the source from incognito mode in chrome to see if you still find the div. — Hooked
– Hooked, Commented Jul 23, 2014 at 13:34

alexislg · Accepted Answer · 2014-07-23 12:37:52Z

2

Maybe you don't have the same page as us on your browser (because you are logged in or so).

When I look at the webpage you provided with Google Chrome, there is nowhere 'answer_count' in the source code. So if Google chrome doen't find it, BeautifulSoup won't either

answered Jul 23, 2014 at 12:37

alexislg

1,01614 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

alexislg Over a year ago

I woulds suggest to use the python 'requests' library. You'll be enable to log in from within your script to any website

Collectives™ on Stack Overflow

error in using BeautifulSoup python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related