NumPy: Improperly Creating 2-D Array

Question

I'm reading in data and trying to create a NumPy array of shape (194, 1). So it should look like: [[4], [0], [9], ...]

I'm doing this:

def parse_data(file_name):
    data = []
    target = []
    with open(file_name) as f:
        for line in f:
            temp = line.split()
            x = [float(x) for x in temp[:2]]
            y = float(temp[2])
            data.append(np.array(x))
            target.append(np.array(y))
    return np.array(data), np.array(target)

x, y = parse_data("data.txt")

when I inspect y.shape, it's (194,), not (194,1) as I expected.

The x has shape (194,2) as I'd expect, however.

Any idea what I'm doing incorrectly?

Thanks!

Can you provide some lines from data.txt?

Daniele Cappuccio
– Daniele Cappuccio

2018-04-12 20:22:42 +00:00
Commented Apr 12, 2018 at 20:22 — Daniele Cappuccio
– Daniele Cappuccio, Commented Apr 12, 2018 at 20:22

user2357112 · Accepted Answer · 2018-04-12 20:27:28Z

3

You seem to have expected np.array(y) to automatically turn your scalar y into a 1-element row. That's not how NumPy works.

np.array(y) is 0-dimensional. Putting a bunch of those in a list and calling array on the list produces a 1-dimensional result, not a 2-dimensional one.

answered Apr 12, 2018 at 20:27

user2357112

286k32 gold badges490 silver badges569 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Primusa · Accepted Answer · 2018-04-12 20:49:42Z

1

When np.array() is called on a list of numpy arrays built from scalars it concatenates them and then creates a numpy array, giving you your (194,) shape.

You can always reshape y to your desired shape:

def parse_data(file_name):
    data = []
    target = []
    with open(file_name) as f:
        for line in f:
            temp = line.split()
            x = [float(x) for x in temp[:2]]
            y = float(temp[2])
            data.append(np.array(x))
            target.append(y)
    return np.array(data), np.array(target).reshape(-1, 1)

x, y = parse_data("data.txt")

Of course you can also fix your problem with:

target.append(np.array([y]))

An example of the behavior I stated above:

import numpy as np
a = np.array(5)
b = np.array(4)
v = [a, b]
v
>>>[array(5), array(4)]
np.array(v)
>>>array(5, 4) #arrays are concatenated

edited Apr 12, 2018 at 20:49

answered Apr 12, 2018 at 20:22

Primusa

13.5k3 gold badges36 silver badges56 bronze badges

3 Comments

ColonelFazackerley Over a year ago

return np.array(data), np.array(target).reshape(-1, 1) likely to be better, in case the amount of data varies.

ColonelFazackerley Over a year ago

I think it should be (-1, 1), not (1, -1).

Christian Over a year ago

You could also use return np.array(data), np.vstack(target) A bit more concise and works as long as target is 1D.

hpaulj · Accepted Answer · 2018-04-12 20:41:40Z

I'd skip the np.array in the iteration.

def parse_data(file_name):
    data = []
    target = []
    with open(file_name) as f:
        for line in f:
            temp = line.split()
            x = [float(x) for x in temp[:2]]
            y = float(temp[2])
            data.append(x)
            target.append(y)
    return np.array(data), np.array(target)

This would create data like:

 [[1.0, 2.0],[3.0, 4.0], ....]

and target like

 [1.2, 3.2, 3.1, ...]

np.array(data) then turns the list of lists into a 2d array, and the list of numbers into a 1d array.

It is then easy to reshape or add a dimension to the 1d, making it (1,n) or (n,1) or what ever you need.

Remember the basic array construction methods are:

np.array([1,2,3])             # 1d
np.array([[1,2],[3,4]])       # 2d

Collectives™ on Stack Overflow

NumPy: Improperly Creating 2-D Array

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related