Slow for loop in python

Question

I have an api which is supposed to get me the venue details from a set of events. There are around 10000 events and the whole process to generate the list is taking around 6.5 seconds because of which the server cpu usage shoots up to 100% and loading of the page is slow. I am using mongodb. Here is the function:

def get_venue():
events_all = Event.objects.all()
locality = ""
venue = []
data = ({'success':False, 'venue': venue})
for event in events_all:
    venue_json = {'venue_name':"", 'local':"", 'locat_id':""}
    try:
        try:
            venue_json['venue_name'] = event.location.location_name
        except Exception as e:
            venue_json['venue_name'] = ""
        try:
            venue_json['local'] = event.location.parent.location_name
        except Exception as e:
            venue_json['local'] = ""
        venue_json['locat_id'] = event.location.location_id
        venue.append(venue_json)
    except Exception as e:
        continue
if len(venue) > 0:
    data['success'] = True
return json.dumps(data)

If I remove the loop then the page loads in under a second. Is there any way to optimize this and bring down the loop time to minimal?

The main way would be to limit the amount of events you're parsing. Just use a slice like for event in events_all[:100] to only the first 100 events. You could alter the number or whether it's the events at the start or end of the list. Apart from that, you're initialising all the values as empty at the start of the loop, but they all get set after that anyway. You may as well initialise an empty dictionary to save a bit of time. — SuperBiasedMan
– SuperBiasedMan, Commented Nov 30, 2015 at 12:26
I have to parse all the events at once because all of the events hold a venue details. And i have to display all the venues as dropdown in select tag. The only problem is that it is taking too much time to go over all the events. Any other suggestions? — Arpit Agarwal
– Arpit Agarwal, Commented Nov 30, 2015 at 12:51
do all event items have location as attributes, and do all location items have parent as attributes? — zehnpaard
– zehnpaard, Commented Nov 30, 2015 at 12:55
Well some event has and some not. So, instead of hitting db to check whether that attribute exist or not, i chose to use try except block. As i thot hitting to db will slow it further down.. — Arpit Agarwal
– Arpit Agarwal, Commented Nov 30, 2015 at 12:58
difficult to say without being sure exactly what your data looks like, but at the very least, bringing venue_json['locat_id'] = event.location.location_id to be immediately beneath your first try is likely to bring the running time down somewhat. — zehnpaard
– zehnpaard, Commented Nov 30, 2015 at 13:01

user2390182 · Accepted Answer · 2015-11-30 13:50:05Z

1

With select_related you can avoid extra database lookups in the for loop (see the docs: https://docs.djangoproject.com/en/1.8/ref/models/querysets/#select-related):

events_all = Event.objects.all().select_related('location__parent')

edited Nov 30, 2015 at 13:50

answered Nov 30, 2015 at 13:41

user2390182

73.7k6 gold badges71 silver badges95 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Arpit Agarwal Over a year ago

I am using flask. Is it supported in flask also?

user2390182 Over a year ago

Idk, this is Django syntax, but there should to be similar functionality in flask:stackoverflow.com/questions/27983129/…

Collectives™ on Stack Overflow

Slow for loop in python

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related