Python: Elegant way to combine output variables of a function run many times

Question

I have a function that returns many output arrays of varying size.

arr1,arr2,arr3,arr4,arr5, ... = func(data)

I want to run this function many times over a time series of data, and combine each output variable into one array that covers the whole time series.

To elaborate: If the output arr1 has dimensions (x,y) when the function is called, I want to run the function 't' times and end up with an array that has dimensions (x,y,t). A list of 't' arrays with size (x,y) would also be acceptable, but not preferred.

Again, the output arrays do not all have the same dimensions, or even the same number of dimensions. Arr2 might have size (x2,y2), arr3 might be only a vector of length (x3). I do not know the size of all of these arrays before hand.

My current solution is something like this:

arr1 = []
arr2 = []
arr3 = []
...

for t in range(t_max):
   arr1_t, arr2_t, arr3_t, ... = func(data[t])

   arr1.append(arr1_t)
   arr2.append(arr2_t)
   arr3.append(arr3_t)
...

and so on. However this is inelegant looking when repeated 27 times for each output array.

Is there a better way to do this?

So, each returned array always has the same shape? For example, given a value passed into func, the returned arr1_t.shape is always the same? — Dunes
– Dunes, Commented Aug 24, 2016 at 20:11

tobias_k · Accepted Answer · 2016-08-24 19:52:03Z

2

You can just make arr1, arr2, etc. a list of lists (of vectors or matrices or whatever). Then use a loop to iterate the results obtained from func and add them to the individual lists.

arrN = [[] for _ in range(N)]  # N being number of results from func
for t in range(t_max):
    results = func(data[t])
    for i, res in enumerate(results):
        arrN[i].append(res)

The elements in the different sub-lists do not have to have the same dimensions.

answered Aug 24, 2016 at 19:52

tobias_k

83.1k12 gold badges130 silver badges186 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

ShadowRanger · Accepted Answer · 2016-08-24 20:10:36Z

Not sure if it counts as "elegant", but you can build a list of the result tuples then use zip to group them into tuples by return position instead of by call number, then optionally map to convert those tuples to the final data type. For example, with numpy array:

from future_builtins import map, zip  # Only on Python 2, to minimize temporaries
import numpy as np

def func(x):
     'Dumb function to return tuple of powers of x from 1 to 27'
     return tuple(x ** i for i in range(1, 28))

# Example inputs for func
data = [np.array([[x]*10]*10, dtype=np.uint8) for  in range(10)]

# Output is generator of results for each call to func
outputs = map(func, data)

# Pass each complete result of func as a positional argument to zip via star
# unpacking to regroup, so the first return from each func call is the first
# group, then the second return the second group, etc.
positional_groups = zip(*outputs)

# Convert regrouped data (`tuple`s of 2D results) to numpy 3D result type, unpack to names
arr1,arr2,arr3,arr4,arr5, ...,arr27 = map(np.array, positional_groups)

If the elements returned from func at a given position might have inconsistent dimensions (e.g. one call might return 10x10 as the first return, and another 5x5), you'd avoid the final map step (since the array wouldn't have consistent dimensions and just replace the second-to last step with:

arr1,arr2,arr3,arr4,arr5, ...,arr27 = zip(*outputs)

making arr# a tuple of 2D arrays, or if the need to be mutable:

arr1,arr2,arr3,arr4,arr5, ...,arr27 = map(list, zip(*outputs))

to make them lists of 2D arrays.

Dunes · Accepted Answer · 2016-08-24 20:48:19Z

This answer gives a solution using structured arrays. It has the following requirement: Ggven a function f that returns N arrays, and the size of each of the returned arrays can be different -- then for all results of f, len(array_i) must always be same. eg.

arrs_a = f("a")
arrs_b = f("b")
for sub_arr_a, sub_arr_b in zip(arrs_a, arrs_b):
   assert len(sub_arr_a) == len(sub_arr_b)

If the above is true, then you can use structured arrays. A structured array is like a normal array, just with a complex data type. For instance, I could specify a data type that is made up of one array of ints of shape 5, and a second array of floats of shape (2, 2). eg.

# define what a record looks like
dtype = [
    # tuples of (field_name, data_type)
    ("a", "5i4"), # array of five 4-byte ints
    ("b", "(2,2)f8"), # 2x2 array of 8-byte floats
]

Using dtype you can create a structured array, and set all the results on the structured array in one go.

import numpy as np

def func(n):
    "mock implementation of func"
    return (
        np.ones(5) * n,
        np.ones((2,2))* n
    )

# define what a record looks like
dtype = [
    # tuples of (field_name, data_type)
    ("a", "5i4"), # array of five 4-byte ints
    ("b", "(2,2)f8"), # 2x2 array of 8-byte floats
]

size = 5
# create array
arr = np.empty(size, dtype=dtype)
# fill in values
for i in range(size):
    # func must return a tuple
    # or you must convert the returned value to a tuple
    arr[i] = func(i)

# alternate way of instantiating arr
arr = np.fromiter((func(i) for i in range(size)), dtype=dtype, count=size)

# How to use structured arrays
# access individual record
print(arr[1]) # prints ([1, 1, 1, 1, 1], [[1, 1], [1, 1]])
# access specific value -- get second record -> get b field -> get value at 0,0
assert arr[2]['b'][0,0] == 2
# access all values of a specific field
print(arr['a']) # prints all the a arrays

Collectives™ on Stack Overflow

Python: Elegant way to combine output variables of a function run many times

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related