Multiple Linear regression and selection of columns error

Question

My problem is that when I tried to fit the model I got this error. I don't know what caused this error but probably the selection of independent variables is not correct. Here's the error

ValueError: Found input variables with inconsistent numbers of samples: [104, 26]

Here's the code I built so far

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats

# Import Excel File
data = pd.read_excel("C:\\Users\\AchourAh\\Desktop\\Multiple_Linear_Regression\\SP Level Reasons Excels\\SP00105485_PL22_AAB_05_09_2018_Reasons.xlsx",'Sheet1') #Import Excel file

# Replace null values of the whole dataset with 0
data1 = data.fillna(0)
print(data1)

# Extraction of the independent and dependent variable
X = data1.iloc[0:len(data1),[0,1,2,3]].values.reshape(-1, 1) #Extract the column of the COPCOR SP we are going to check its impact
Y = data1.iloc[0:len(data1),4].values.reshape(-1, 1) #Extract the column of the PAUS SP
print(X)
print(Y)

# Importing
from sklearn.linear_model import LinearRegression
from sklearn import model_selection

# Fitting a Linear Model
lm = LinearRegression() #create an lm object of LinearRegression Class
lm.fit(X, Y)
plt.scatter(X, Y, color = 'red')#plots scatter graph of COP COR against PAUS for values in X_train and y_train
plt.plot(X, lm.predict(X), color = 'blue')#plots the graph of predicted PAUS against COP COR.
plt.title('SP000905974')
plt.xlabel('COP COR Quantity')
plt.ylabel('PAUS Quantity')
plt.show()#Show the graph

The first columns of my excel file contains the independent variables and the 4th column contains the dependent variable. I have another code of Simple linear regression that works fine but when I tried to apply a Multiple Linear regression I just changed this line but I don't what I did wrong.

  X = data1.iloc[0:len(data1),[0,1,2,3]].values.reshape(-1, 1)

to notice, I am a beginner with this.

Why are you reshaping the X and Y by using reshape(-1, 1)? Do you know what it does? — Vivek Kumar
– Vivek Kumar, Commented Oct 10, 2018 at 8:34
yes to be able to plot X and Y later. Otherwise another error will raise : ValueError: x and y must be the same size when plotting — user10384553
– user10384553, Commented Oct 10, 2018 at 8:43

Florian H · Accepted Answer · 2018-10-10 10:10:47Z

0

your problem is indeed the reshaping of X.

Example:

pd.DataFrame([[1,2],[3,4],[5,6]], columns = ["a", "b"]).values

is a numpy array that looks like

array([[1, 2],
       [3, 4],
       [5, 6]], dtype=int64)

while

pd.DataFrame([[1,2],[3,4],[5,6]], columns = ["a", "b"]).values.reshape(-1,1)

doubles your amount of rows (because of two columns are reshaped into one)

array([[1],
       [2],
       [3],
       [4],
       [5],
       [6]], dtype=int64)

So in your case after reshaping your four columns into one you have four times more rows in X than in Y, while lm.fit(X, Y) needs you to have the same amount of rows in X and Y.

answered Oct 10, 2018 at 10:10

Florian H

3,0722 gold badges16 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user10384553 Over a year ago

ok but when I delete the reshape and try to scatter plot I receive another error : ValueError: x and y must be the same size

Florian H Over a year ago

thats because your X Data has four Dimensions (Columns) and a scatter plot is two dimensional. your scatter plot doesnt make any sense if you reshape your data like you did. The way to go here would be four scatter plots - one X Column and your Y column in each plot.

Collectives™ on Stack Overflow

Multiple Linear regression and selection of columns error

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related