Fastest way to process images from web in python?

Question

I'm using urllib2, cstringIO and PIL. I need to really tune this and make it very fast (at least half the current speed)

I access and load the image using the below.

imageurl = "http://bit.ly/wOqVTE"

@log_performance
def get_image(imageurl):
    img_file = urllib.urlopen(imageurl)
    data = StringIO(img_file.read())
    im = Image.open(data)
    size = 128, 128
    im.thumbnail(size, Image.ANTIALIAS)
    return im

Then process the image using:

@log_performance
def process_image(image, sample_limit=10000, top=10):
    colors = image.getcolors(sample_limit)
    sc = sorted(colors, key=lambda x: x[0], reverse=True)
    return sc[:top]

This takes on average 0.6 seconds to get the image and around 0.006 seconds to process.

How can I speed up the get and load process?

The full gist can be found here. https://gist.github.com/1920167

>>>>Function: get_image, Executed:20, Avg Time:0.558275926113
>>>>Function: process_image, Executed:20, Avg Time:0.00609920024872

I will add bounty of 50 for anyone that can half the time.

Try splitting get_image up to see how much time is being spent on network I/O and how much is being spent on PIL. — icktoofay
– icktoofay, Commented Feb 27, 2012 at 0:51
What @icktoofay said. Are you sure you're not just hitting the network limits? If this is a server response time issue you could try splitting the image fetches out across a multiprocessing.Pool to get a few concurrent downloads. — Katriel
– Katriel, Commented Feb 27, 2012 at 1:00

WooParadog · Accepted Answer · 2012-02-27 01:07:59Z

2

Since it's getting the images that takes the longest time, why not use threading(or Gevent) to get those images concurrently, throw the results in a task queue, and process when they are ready.

And add cache for images with the same url...

answered Feb 27, 2012 at 1:07

WooParadog

6494 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

ciphor · Accepted Answer · 2012-02-27 04:00:22Z

0

Since the image fetching depends on the network I/O speed, you should use asynchronous I/O to enhance the overall performance.

Reference: http://sourceforge.net/projects/asynchttp/

answered Feb 27, 2012 at 4:00

ciphor

8,31811 gold badges57 silver badges71 bronze badges

Collectives™ on Stack Overflow

Fastest way to process images from web in python?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related