0 
a = [(1,2),(3,1),(4,4),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5),(5,5)]
# Quite a lot tuples in the list, 6 digits~
# I want to split it into rows and columns.
rows = 5
cols = 5

Data structure is
rows and cols are the index for the bit list
[rows, cols, (data)]

I use loop to do this, but it takes too long for processing a big amount of tuples.

processed_data = []
index = 0
for h in range(0, rows - 1):
    for w in range(0, cols - 1):
        li = []
        li = [h, w, a[index]]
        processed_data.append(li)
        index += 1

This operation takes too long, is there a way to do optimization? Thanks very much!

Improve this question
6
  • Any thoughts on approach? What have you tried so far? Why are you optimizing this in particular - are you sure it is the bottleneck in your program (i.e., what percent of the execution time is it taking, and what is your desired target) Commented Oct 23, 2010 at 2:27
  • @Merlyn Morgan-Graham, it needs to be optimized even if it's not a bottleneck. Creating an empty list just to immediately replace the binding with another brand new list? Not good. Commented Oct 23, 2010 at 2:30
  • @aaronasterling: What you just described sounds like code stink, not an optimization. They are entirely different things, and require different approaches. Code stink you must learn to identify, and avoid creating. Optimizations should only be done once balancing bottlenecks that you can't ship with (that you identified w/ proper tools and realistic scenarios) vs ease of reading the code/ease of future maintenance. Commented Oct 23, 2010 at 2:35
  • 1
    Why a 2-d data structure, and not a dictionary mapping a tuple to a total count such as: {(1,2) : 123, (4,5) : 534 ...}? What about the 6 digits? What are you really trying to accomplish here? What is this for, mate? Commented Oct 23, 2010 at 2:42
  • @aaronasterling: Although you're right, sometimes you will go "why did I do it that way!" and fix it w/o bothering to profile :) Commented Oct 23, 2010 at 3:19

3 Answers 3

Reset to default
2

It's not at all clear to me what you want but here's a shot at the same loop in a more optimized manner:

import itertools as it

index = it.count(0) 
processed_data = [[h, w, a[next(index)]] 
                 for h in xrange(0, rows - 1)
                 for w in xrange(0, cols - 1)]

or, since you've already imported itertools, 

index = ite.count(0)
indices = it.product(xrange(0, rows-1), xrange(0, cols-1))
processed_data = [[h, w, a[next(index)]] for h, w in indices]

The reason that these are faster is that they use list comprehensions instead of for loops. List comprehensions have their own opcode, LIST_APPEND, which routes directly to the append method on the list that's being constructed. In a normal for loop, the virtual machine has to go through the whole processes of looking up the append method on the list object which is fairly pricey.

Also, itertools is implemented in C so if it's not faster for the same algorithm, then there's a bug in itertools.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Yes, it's about 3 seconds shorter!
@user469652, how long was it taking before?
2

Fine, if you really want the indices that badly...

[divmod(i, cols) + (x,) for i, x in itertools.izip(itertools.count(), a)]
Improve this answer

Comments

0

Sounds like you want to split it into evenly-sized chunks.

Improve this answer

4 Comments

no, that shouldn't be right because only one element of the old list ends up in each element of the new list in the code given.
@aaronsterling: The only difference is that it doesn't include h and w. But this isn't necessarily a problem since you can derive them from where you are in the nested structure via enumerate().
no, the difference is that slices of the list are being returned by the generator you linked too because OP of that question wanted chunks of data. OP of this question wants essentially the same list decorated with additional indices.
@aaronsterling: The adornments can be derived on the other side. Yes, it means a little more work on the other side, but it means less work on this side.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.