Apply function to single column of structured numpy array in Python

Question

I have a structured numpy array with two columns. One column contains a series of date times as strings, and the other contains measured values corresponding to that date.

data = array([('date1', 2.3), ('date3', 2.4), ...] 
             dtype=[('date', '<U16'), ('val', '<f8')])

I also have a number of functions similar to the following:

def example_func(x):
    return 5*x + 1

I am trying to apply example_func to the second column of my array and generate the result

array([('date1', 12.5), ('date3', 11.6), ...] 
      dtype=[('date', '<U16'), ('val', '<f8')])

Everything I try, however, either raises a future warning from numpy or requires a for loop. Any ideas on how I can do this efficiently?

To clarify, I want to modify the existing array so that I retain knowledge about which date corresponds with which value in example_func(data['val']). I guess I could use your suggestion and then create a new array using np.array(data['date'], example_func(data['val'])). However, I was hoping to do this without creating a new array. — user144153
– user144153, Commented Apr 2, 2017 at 20:00
To modify the existing array, simply assign : data['val'] = ..? — Divakar
– Divakar, Commented Apr 2, 2017 at 20:01
This raise a FutureWarning from numpy. To quote the numpy developers "...This code will likely break in a future numpy release..." — user144153
– user144153, Commented Apr 2, 2017 at 20:03

hpaulj · Accepted Answer · 2017-04-02 20:49:02Z

2

This works for me:

In [7]: example_func(data['val'])
Out[7]: array([ 12.5,  13. ])
In [8]: data['val'] = example_func(data['val'])
In [9]: data
Out[9]: 
array([('date1',  12.5), ('date3',  13. )], 
      dtype=[('date', '<U16'), ('val', '<f8')])
In [10]: np.__version__
Out[10]: '1.12.0'

I have gotten future warnings when accessing several fields (with a list of names), and then attempting some sort of modification. It suggests making a copy etc. But I can't generate such a warning with a single field access like this.

In [15]: data[['val', 'date']]
Out[15]: 
array([( 12.5, 'date1'), ( 13. , 'date3')], 
      dtype=[('val', '<f8'), ('date', '<U16')])
In [16]: data[['val', 'date']][0] = (12, 'date2')
/usr/local/bin/ipython3:1: FutureWarning: Numpy has detected that you (may be) writing to an array returned
by numpy.diagonal or by selecting multiple fields in a structured
array. This code will likely break in a future numpy release --
see numpy.diagonal or arrays.indexing reference docs for details.
The quick fix is to make an explicit copy (e.g., do
arr.diagonal().copy() or arr[['f0','f1']].copy()).

Developers aren't happy with how they access several fields at once. It's ok to read them, but changing is under evaluation. And in '1.13' there's some change about copying fields by position rather than name.

edited Apr 2, 2017 at 20:49

answered Apr 2, 2017 at 20:42

hpaulj

233k14 gold badges260 silver badges392 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user144153 Over a year ago

This is interesting because for me data['val'] = example_func(data['val']) raises a FutureWarning. I wonder if it has to do with the context in which I am using it IRL (my arrays are all in nested dictionaries). Either way, thanks for the insight. I'll most likely continue with this approach.

hpaulj Over a year ago

I don't see how arrays in dictionaries should make a difference. If you can construct a minimal example that reproduces the warning, I'll look at it.

user144153 Over a year ago

Your right, it doesn't make a difference. I've tried to isolate the behavior and can't create a simplified test case that reproduces the behavior outside the overall code. I'll tinker with it and post an update if / when I figure it out.

Collectives™ on Stack Overflow

Apply function to single column of structured numpy array in Python

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related