python linear regression implementation

Question

I've been trying to do my own implementation of a simple linear regression algorithm, but I'm having some trouble with the gradient descent.

Here's how I coded it:

def gradientDescentVector(data, alpha, iterations):
    a = 0.0
    b = 0.0
    X = data[:,0]
    y = data[:,1]
    m = data.shape[0]
    it = np.ones(shape=(m,2))
    for i in range(iterations):
        predictions = X.dot(a).flatten() + b

        errors_b = (predictions - y)
        errors_a = (predictions - y) * X

        a = a - alpha * (1.0 / m) * errors_a.sum()
        b = b - alpha * (1.0 / m) * errors_b.sum()
    return a, b

Now, I know this won't scale well with more variables, but I was just trying with the simple version first, and follow up from there.

I was following the gradient descent algorithm from the machine learning course at coursera:

But I'm getting infinite values after ~90 iterations (on a specific dataset), and haven't been able to wrap my head around this so far.

I've tried iterating over each value before I learned about numpy's broadcasting and was getting the same results.

If anyone could shed some light on what could be the problem here, it would be great.

Nagabhushan Baddi · Accepted Answer · 2016-09-14 11:01:59Z

1

It is clear that the parameters are diverging from the optimum ones. One possible reason may be that you are using too large a value for the learning rate ("alpha"). Try decreasing the learning rate. Here is a rule of thumb. Start always from a small value like 0.001. Then try increasing the learning rate by taking a three time higher learning rate than previous. If it gives less MSE error (or whatever error function you are using), then its fine. If not try taking a value between 0.001 and 0.003. Next if the latter hold true, then try this recursively until you reach a satisfying MSE.

answered Sep 14, 2016 at 11:01

Nagabhushan Baddi

1,20411 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

msk Over a year ago

I had already tried to 0.001 and it was not working, but I went to try again with an even lower learning rate, 0.0000001, and it started working properly, thanks for that! Tho I wasn't using any kind of error function, maybe I'll try to look into it after.

Collectives™ on Stack Overflow

python linear regression implementation

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related