Error while trying to parse a website url using python . how to debug it?

Question

#!/usr/bin/python
import json
import urllib
from BeautifulSoup import BeautifulSoup
from BeautifulSoup import BeautifulStoneSoup
import BeautifulSoup  


def showsome(searchfor):
  query = urllib.urlencode({'q': searchfor})
  url = 'http://ajax.googleapis.com/ajax/services/search/web?v=1.0&%s' % query
  search_response = urllib.urlopen(url)
  search_results = search_response.read()
  results = json.loads(search_results)
  data = results['responseData']
  print 'Total results: %s' % data['cursor']['estimatedResultCount']
  hits = data['results']
  print 'Top %d hits:' % len(hits)
  for h in hits: 
    print ' ', h['url']
    resp = urllib.urlopen(h['url'])
        res = resp.read()

    soup = BeautifulSoup(res)
        print soup.prettify()
  print 'For more results, see %s' % data['cursor']['moreResultsUrl']

showsome('sachin')

What is the wrong in this code ?

Note all the 4 links that I am getting out of the search , I am feeding it back to extract the contents out of it , and then use BeautifulSoup to parse it . How should I go about it ?

I don't know whats wrong. What's wrong? Doesn't it work? How about explaining what the problem is? :-) — Lennart Regebro
– Lennart Regebro, Commented Dec 21, 2010 at 7:16
I want to parse the content of the of url which I am calling , and the reponse that is stored in the variable "resp" . So the question is once I have the response how Do I parse the contents of it ? — Hick
– Hick, Commented Dec 21, 2010 at 7:18
You said "Error while trying to...". What was the exact error? — Laurence Gonsalves
– Laurence Gonsalves, Commented Dec 21, 2010 at 7:20

Laurence Gonsalves · Accepted Answer · 2010-12-21 07:11:18Z

1

What is the wrong in this code ?

Your indentation is all wonky in the for loop, and this line:

import BeautifulSoup

should be deleted, as it masks this earlier import:

from BeautifulSoup import BeautifulSoup

answered Dec 21, 2010 at 7:11

Laurence Gonsalves

144k38 gold badges264 silver badges315 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Christopher Toth · Accepted Answer · 2010-12-21 07:37:47Z

0

It appears you're trying to access the Google ajax API in Python. You might consider using the Xgoogle library available on Github: https://github.com/pkrumins/xgoogle

answered Dec 21, 2010 at 7:37

Christopher Toth

314 bronze badges

Comments

mariana soffer · Accepted Answer · 2010-12-21 08:27:32Z

0

You can debug python code in a proper environment that lets you examine what is going on with the variables, stacks and all using the eclipse ide with a module name pydev, try it to see what is going on inside.

answered Dec 21, 2010 at 8:27

mariana soffer

1,85312 silver badges17 bronze badges

Collectives™ on Stack Overflow

Error while trying to parse a website url using python . how to debug it?

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related