Create NumPy array from list with operations

Question

I have data in a Python list from an SQLite database in the following format:

# This is an example
data = [(1, '12345', 1, 0, None), (1, '34567', 1, 1, None)]

From this list of tuples I want to create a 2D NumPy array, converting each tuple to an array. I also want the values at index 1 in the tuples converted from string to numbers, and the values at last index converted to 0 if None, 1 otherwise.

What it should look like afterwards:

transformed_data = np.asarray([[1, 12345, 1, 0, 0], [1, 34567, 1, 1, 0]])

I am able to with simple for loops, however I'd like a more Pythonic solution with native NumPy methods or otherwise. I am working with a very large database, so complexity matters.

"While doing so, I also want to be able to specify transformations on the data." I don't think you can do this all in one step. Do the transformations as a separate step. — Code-Apprentice
– Code-Apprentice, Commented Sep 9, 2022 at 14:53

mozway · Accepted Answer · 2022-09-09 14:38:01Z

2

pandas is quite good at this:

import pandas as pd
                      # set up DataFrame
transformed_data = (pd.DataFrame(data)
                      # convert to numeric
                      .apply(pd.to_numeric, errors='coerce')
                      # replace null with 0
                      # trying to cast as integer if possible
                      .fillna(0, downcast='infer')
                      # convert to numpy array
                      .to_numpy()
                   )

output:

array([[    1, 12345,     1,     0,     0],
       [    1, 34567,     1,     1,     0]])

answered Sep 9, 2022 at 14:38

mozway

267k13 gold badges56 silver badges106 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Prins · Accepted Answer · 2022-09-09 14:51:00Z

1

If your tuple is small and of a fixed size then you can use a list comprehension:

result = [(a, int(b), c, d, 0 if e is None else e) for a, b, c, d, e in data]

Or a little shorter:

result = [(d[0], int(d[1]), *d[2:4], d[4] if d[4] else 0) for d in data]

edited Sep 9, 2022 at 14:51

answered Sep 9, 2022 at 14:45

Prins

1,0511 gold badge6 silver badges9 bronze badges

1 Comment

Guillaume Over a year ago

This would work, however my tuples have over 500 elements so it would make the code quite convoluted

Collectives™ on Stack Overflow

Create NumPy array from list with operations

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related