Trying to improve efficiency of iterative Python program

Question

Good morning, and sorry if this is a vague question. I'll try to be as descriptive as possible.

Basically, I am using a Python code to post-process the results from an air dispersion model to test out different scenarios. I am using Python as it can iterate through the results in a matter of seconds while the dispersion software takes hours. My problem is that the code will still take weeks to run through all my scenarios and I'm wondering if it's due to poor programming. I won't put the whole code here, as a lot if it is not relevant, but I will go through the steps I'm taking. First, here's an outline of my problem:

I have 17 sources all acting simultaneously
Each source can have four different emission rates, which are independant of the other source. I.e. source #1 can have emission rate a, b, c, or d. As can source #2-#17.
Each source can take one of two states. We will call them working or not working, but in both states there is an emissions rate. HOWEVER, only 5 sources can be working simultaneously. This is important.

To summarize, the emission from each source is a function of the four emission rates as well as the two states. So each source has 8 possible emission scenarios, and all 17 sources could be in any of these scenarios at a time. Quite a lot of permutations!

Here's how I'm currently computing the results. I want to know for each combination of states what the maximum result would be. If you're familiar with air dispersion modelling, I have already calculated the results based on a 1 g/s emission rate, so I can scale the results by the emission rates above.

THE CODE:

sources = ['1','2',...'17']   
emission_rates = ['a','b','c','d'] 
Source_1_results = [list of values of length x] ## NOTE THAT x is VERY LONG.  THESE ARE HUGE ARRAYS (400,000 values)
Source_2_result = [list of values of length x]
.
.
Source_17_results = [list of values of length x]

working_sources = list(itertools.combinations(sources, 5))
source_emission_rate = list(itertools.combinations_with_replacement(emission_rates, 17))

for e in source_emission_rate:
    for w in working_sources:
        temp_results = []
        for num, source in enumerate(sources):
            temp_results[num] = [Source_x_result * e * w] ##THIS LINE INVOLVES SOME LOOKUP IN MY CODE TO REFERENCE THE ACTUAL RESULTS AND EMISSIONS ETC.

I'm sorry if this isn't enough code. I can post the full code, but again, for the most part it's just assigning variables etc.

My question is: is there a quicker way to iterate through all my possible states? My code is currently working, but I have limited python knowledge and would like to be able to run it more frequently while changing variables etc.

Thank you in advance!

Why do you turn the result of itertools.combinations into a list? the generator is enough in that case. — serge-sans-paille
– serge-sans-paille, Commented Sep 5, 2015 at 15:15

serge-sans-paille · Accepted Answer · 2015-09-05 15:56:59Z

1

this should go slightly faster (less intermediate list + list comprehension)

working_sources = itertools.combinations(sources, 5)
source_emission_rate = itertools.combinations_with_replacement(emission_rates, 17)

for e in source_emission_rate:
    for w in working_sources:
        temp_results = [source * e * w for source in sources]

answered Sep 5, 2015 at 15:56

serge-sans-paille

2,13913 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Trying to improve efficiency of iterative Python program

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related