Using httplib2 in python 3 properly? (Timeout problems)

Question

Hey, first time post, I'm really stuck on httplib2. I've been reading up on it from diveintopython3.org, but it mentions nothing about a timeout function. I look up the documentation, but the only thing I see is an ability to put a timeout int but there are no units specified (seconds? milliseconds? What's the default if None?) This is what I have (I also have code to check what the response is and try again, but it's never tried more than once)

h = httplib2.Http('.cache', timeout=None)
for url in list:
    response, content = h.request(url)
    more stuff...

So the Http object stays around until some arbitrary time, but I'm downloading a ton of pages from the same server, and after a while, it hangs on getting a page. No errors are thrown, the thing just hangs at a page. So then I try:

h = httplib2.Http('.cache', timeout=None)
for url in list:
    try:
        response, content = h.request(url)
    except:
        h = httplib2.Http('.cache', timeout=None)
    more stuff...

But then it recreates another Http object every time (goes down the 'except' path)...I dont understand how to keep getting with the same object, until it expires and I make another. Also, is there a way to set a timeout on an individual request?

Thanks for the help!

Bengt · Accepted Answer · 2013-12-19 17:13:38Z

4

Due to bug httplib2 measured the timeout in seconds multiplied by 2 until version 0.7.5 (2012-08-28).

edited Dec 19, 2013 at 17:13

answered Jul 24, 2012 at 16:11

Bengt

14.6k7 gold badges53 silver badges67 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Cyril N. Over a year ago

The bug is now closed

Lennart Regebro · Accepted Answer · 2010-01-30 08:46:52Z

1

Set the timeout to 1, and you'll pretty quickly know if it means one millisecond or one second.

I don't know what your try/except should solve, if it hangs on h.request(url) in one case it should hang in the other.

If you run out of memory in that code, then httplib2 doesn't get garbage collected properly. It may be that you have circular references (although it doesn't look like it above) or it may be a bug in httlib2.

answered Jan 30, 2010 at 8:46

Lennart Regebro

173k45 gold badges230 silver badges254 bronze badges

Comments

Gino Mempin · Accepted Answer · 2024-10-17 23:19:00Z

0

The httplib2 timeout unit is in seconds.

edited Oct 17, 2024 at 23:19

Gino Mempin

30.5k31 gold badges125 silver badges174 bronze badges

answered Oct 17, 2024 at 18:27

ddf

1

Collectives™ on Stack Overflow

Using httplib2 in python 3 properly? (Timeout problems)

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related