gradient descent using python numpy matrix class

Question

I'm trying to implement the univariate gradient descent algorithm in python. I have tried a bunch of different ways and nothing works. What follows is one example of what I've tried. What am I doing wrong? Thanks in advance!!!

from numpy import *

class LinearRegression:

  def __init__(self,data_file):
    self.raw_data_ref = data_file
    self.theta = matrix([[0],[0]])
    self.iterations = 1500
    self.alpha = 0.001


  def format_data(self):
    data = loadtxt(self.raw_data_ref, delimiter = ',')
    dataMatrix = matrix(data)
    x = dataMatrix[:,0]
    y = dataMatrix[:,1]
    m = y.shape[0]
    vec = mat(ones((m,1)))
    x = concatenate((vec,x),axis = 1)
    return [x, y, m]


  def computeCost(self, x, y, m):
    predictions = x*self.theta
    squaredErrorsMat = power((predictions-y),2)
    sse = squaredErrorsMat.sum(axis = 0)
    cost = sse/(2*m)
    return cost


  def descendGradient(self, x, y, m):
      for i in range(self.iterations):

          predictions = x*self.theta
          errors = predictions - y
          sumDeriv1 = (multiply(errors,x[:,0])).sum(axis = 0)
          sumDeriv2 = (multiply(errors,x[:,1])).sum(axis = 0)

          print self.computeCost(x,y,m)

          tempTheta = self.theta
          tempTheta[0] = self.theta[0] - self.alpha*(1/m)*sumDeriv1
          tempTheta[1] = self.theta[1] - self.alpha*(1/m)*sumDeriv2

          self.theta[0] = tempTheta[0]
          self.theta[1] = tempTheta[1]


      return self.theta



regressor = LinearRegression('ex1data1.txt')
output = regressor.format_data()
regressor.descendGradient(output[0],output[1],output[2])
print regressor.theta

A little update; I previously tried to do it in a more "vectorized" way, like so:

def descendGradient(self, x, y, m):
  for i in range(self.iterations):

      predictions = x*self.theta
      errors = predictions - y

      sumDeriv1 = (multiply(errors,x[:,0])).sum(axis = 0)
      sumDeriv2 = (multiply(errors,x[:,1])).sum(axis = 0)

      gammaMat = concatenate((sumDeriv1,sumDeriv2),axis = 0)
      coeff = self.alpha*(1.0/m)
      updateMatrix = gammaMat*coeff
      print updateMatrix, gammaMat


      jcost  = self.computeCost(x,y,m)
      print jcost
      tempTheta = self.theta
      tempTheta = self.theta - updateMatrix
      self.theta = tempTheta

  return self.theta

This resulted in a theta of [[-0.86221218],[ 0.88827876]].

What exactly do you mean by "nothing works?" What happens when you try to run this code? What did you think was wrong, and what have you tried in order to fix the code? — Colin Nichols
– Colin Nichols, Commented May 30, 2013 at 0:39
@Colin, thanks a lot for responding. I don't know why it didn't work but the following occurred to me: extraneous temporary variable storing the value of theta, misplaced return statement, and as per below, not specifying floats. I also tried to implement it in a more "vectorized" way as well, as per my edit. — Eric P
– Eric P, Commented May 30, 2013 at 14:51

score 2 · Accepted Answer · 2013-05-30 03:27:30Z

2

You have two problems, both are related to floating points:

1. Initialize your theta matrix like this:

self.theta = matrix([[0.0],[0.0]])

2. Change the update lines, replacing (1/m) with (1.0/m):

tempTheta[0] = self.theta[0] - self.alpha*(1.0/m)*sumDeriv1
tempTheta[1] = self.theta[1] - self.alpha*(1.0/m)*sumDeriv2

On an unrelated note: your tempTheta variable is unnecessary.

edited May 30, 2013 at 3:27

answered May 30, 2013 at 2:45

user2134086

Sign up to request clarification or add additional context in comments.

2 Comments

Eric P Over a year ago

Thanks jeremy for your tips. The extraneous tempTheta variable had crossed my mind but I wasn't sure about it. Also, using the decimals to tell python to use floats didn't change the outcome. I still haven't gotten it to work... but not for lack of trying. I think theta should converge to [[-3.63029144],[1.16636235]]. Thanks again.

user2134086 Over a year ago

Making the above changes and making self.iterations = 15000 and making self.alpha = 0.01 gave me a result of [[-3.89578088] [ 1.19303364]] (using the ex1data1.txt from the Coursera course).

Collectives™ on Stack Overflow

gradient descent using python numpy matrix class

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related