python threading issue with asynchronous http request

Question

I'm building an asynchronous HTTP request class with urllib2. The code looks like this:

import urllib2, threading, datetime

class ExportHandler(urllib2.HTTPHandler):
            def http_response(self, link, actualdate, nowdate ):
                link += "&export=csv"
                export = urllib2.urlopen( link )

for link, actualdate in commissionSummaryLinks:
            o = urllib2.build_opener(ExportHandler())
            t = threading.Thread(target=o.open, args=(link, actualdate, datetime.datetime.now().strftime("%Y%m%d%H%M%S")))
            t.start()

            print "I'm asynchronous!"
            t.join()

            print "ending threading"

Suffice it to say commissionSummaryLinks IS populated and actualdate is a date time.datetime.strptime() object.

Anyway, I'm receiving an error from all the issued threads that looks like this:

Exception in thread Thread-9:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 808, in __bootstrap_inner
   self.run()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 761, in run
   self.__target(*self.__args, **self.__kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 402, in open
   req = meth(req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1123, in do_request_
   'Content-length', '%d' % len(data))
TypeError: object of type 'datetime.date' has no len()

I'm running this on OS X (if it matters). Can anyone tell me what the problem here is?

Alex Smith · Accepted Answer · 2014-03-31 14:07:34Z

1

When you instantiate your thread, you need to provide the arguments to o.open, not arguments to the http_response method of ExportHandler.

In this case, o.open has the following method signature:

open(self, fullurl, data=None, timeout=<object object>) method of urllib2.OpenerDirector instance

My guess is that you should only need to set args=(link,).

If you still need to use those other arguments, you'll probably want to modify the constructor of ExportHandler to take the other arguments you need, and then use them as appropriate in the http_response method. Take a look at the Python Class tutorial for more information on defining a class constructor.

edited Mar 31, 2014 at 14:07

answered Mar 31, 2014 at 3:27

Alex Smith

1,53512 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

sadmicrowave Over a year ago

Ok, you are right, that seemed to work. However, what if I actually DO need the other arguments for more actions that take place within the ExportHandler class? How would I pass those in?

Alex Smith Over a year ago

@sadmicrowave, I expanded my original answer to address this.

sadmicrowave Over a year ago

awesome, but how do I get printing to work? The function is not doing the things it needs to do, yet I'm getting the output I'm asynchronous! and ending threading but nothing prints from within the http_repsonse method for me to troubleshoot and see how far the script gets...

Alex Smith Over a year ago

@sadmicrowave, there aren't any print statements in the http_response method, so I wouldn't expect anything to be printed unless something was added there.

sadmicrowave Over a year ago

yes, I've added some lines in my code that differ from the OP. Including a file write statement that should be writing files to a directory, but the files are not there after successfully completing the asynchronous requests, which leads me to believe that http_response() isn't being called at all...

Collectives™ on Stack Overflow

python threading issue with asynchronous http request

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related