Replacing loop with List Comprehension instead of loop getting a function to return a new array within the list comprehension

Question

Basically I am trying to avoid looping through big arrays before I had code that looked like this:

for rows in book:
        bs = []
        as = []
        trdsa = []
        trdsb = []
        for ish in book:
            var = (float(str(ish[0]).replace(':',"")) - float(str(book[0]).replace(':',"")))
            if var < .1 and var > 0 :
                bs.append(int(ish[4]))
                as.append(int(ish[5]))
                trdsa.append(int(ish[-2]))
                trdsb.append(int(ish[-1]))
                time = ish[0]
            bflow = sum(numpy.diff(bs))
            aflow = sum(numpy.diff(as))
            OFI = bflow - aflow - sum(trdsb) + sum(trdsa)
            OFIlist.append([time,bidflow,askflow,OFI])

I don't want to loop through the list twice as it consumes way too much time. I was thinking I could do a list comprehension but I'm not sure if I'm on the right track

OFIcreate(x,y):
    bs = []
    as = []
    trdsa = []
    trdsb = []
    var = (float(str(y[0]).replace(':',"")) - float(str(x[0]).replace(':',"")))
    if var < .1 and var >= 0 :
        bs.append(int(ish[4]))
        as.append(int(ish[5]))
        trdsa.append(int(ish[-2]))
        trdsb.append(int(ish[-1]))
        time = ish[0]
    bflow = sum(numpy.diff(bs))
    aflow = sum(numpy.diff(as))
    OFI = bflow - aflow - sum(trdsb) + sum(trdsa)
    OFIlist.append([time,bidflow,askflow,OFI])
    return OFIlist

    OFIc = [ OFIcreate(x,y) for x in book for y in book)

The problem is that I want to loop through the list and group all instances where var >=0 and var <.1 then append values into a new list. The way I have it now I dont think it does that as it will just keep creating lists with a length of one. Any ideas on how I can accomplish this? Or rather how can I make the first block of code more efficient?

you didn't returned anything from OFIcreate(x,y), so OFIc will be just a list of None(s) — Ashwini Chaudhary
– Ashwini Chaudhary, Commented Sep 27, 2012 at 14:40
@AshwiniChaudhary sorry I forgot the return statement but that doesn't solve the problem — Rtrader
– Rtrader, Commented Sep 27, 2012 at 14:56

Pierre GM · Accepted Answer · 2012-09-28 21:14:39Z

1

While list comprehensions are indeed interpreted faster than regular loops, they can't work for everything. I don't think you could replace your main for loop by a list comprehension. However, there might be some room for improvement:

You could build a list of your time by list comprehension.
```
time = [ish[0] for ish in book]
```

You could compute a list of var by list comprehension and transform it a np.array.

var = np.array([t.replace(':',',') for t in time], dtype=float)
var -= float(str(book[0]).replace(":", ","))

You could build 4 numpy int arrays for bs, as (that you need to rename, as is a Python keyword)...
You could then filter your bs... arrays with fancy indexing:
```
bs_reduced = bs[(var < 0.1) & (var >=0)]
```

edited Sep 28, 2012 at 21:14

answered Sep 27, 2012 at 15:14

Pierre GM

20.5k3 gold badges58 silver badges67 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Rtrader Over a year ago

when I try 'bs_reduced = bs[(var < 0.1) and (var >=0)]' i get a valueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() any idea on how I can fix this?

Pierre GM Over a year ago

use np.logical_and(var < 0.1, var >=0) or (var <0.1) & (var >=0): you really need to get a 1D boolean array with the same size as bs for it to work. I gonna correct that.

glglgl · Accepted Answer · 2012-09-27 14:48:26Z

1

I don't want to loop through the list twice as it consumes way too much time. I was thinking I could do a list comprehension but I'm not sure if I'm on the right track

Probably not. A list comprehension does nothing but looping through the given list(s), so it should make no noticeable difference.

answered Sep 27, 2012 at 14:48

glglgl

91.5k13 gold badges157 silver badges230 bronze badges

3 Comments

Rtrader Over a year ago

It would make a big difference list comprehension is way faster than standard looping

glglgl Over a year ago

@user1440194 But if you do a double looping via the same list, you have O(n²).

finity Over a year ago

@glglgl, you make a good point. He's comparing every book to every other book, O(n^2). One way to make the operation faster then is to change that. user1440194, are the books sorted by [0]? If so, perhaps for every book you only need to consider a few other books, instead of the entire list. You'd consider books on either side of a given book until 0 < var < .1 was no longer true. If your book list isn't sorted, perhaps you should do that first. Maybe there's a convenient point to do that. Maybe do a list comprehension to build a (float([0]), index) list, then sort that, then look at books.

Collectives™ on Stack Overflow

Replacing loop with List Comprehension instead of loop getting a function to return a new array within the list comprehension

2 Answers 2

2 Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related