1

I am attempting to make a neural network python module that can be imported form other programs. I am testing this module with some data from kaggle. However, no matter what the inputs are, the outputs is almost always the same, within 1e-7!! I have normalized my data and tried adding more hidden layers and more hidden nodes. I have also lowered my learning rate and made it lower as the training process continues. I tried adding a bias but that had a negative effect

Here is the importable hidden_net.py module:

class network:
    def __init__(self,layer_num,learning_rate=0.7,seed=None,logistic_coefficent=0.9):
        self.logistic_coefficent=logistic_coefficent
        self.learning_rate=learning_rate
        self.w0 = np.random.random((layer_num[0],layer_num[1]))
        self.w1 = np.random.random((layer_num[1],layer_num[2]))

        np.random.seed(seed)
    def sigmoid(self,x,reverse=False):
            if(reverse==True):
                return x*(1-x)
            return 1/(1+np.exp(-x))

    def train(self,inps,outs,mod_learn_rate=0):
        if mod_learn_rate == 0:
            mod_learn_=self.learning_rate
        inps=np.array(inps)
        layer0 = inps
        layer1 = self.sigmoid(np.dot(layer0,self.w0))
        layer2 = self.sigmoid(np.dot(layer1,self.w1))
        layer2_error = outs - layer2
        layer2_delta = layer2_error*self.sigmoid(layer2,reverse=True)#*mod_learn_rate
        layer1_error = layer2_delta.dot(self.w1.T)
        layer1_delta = layer1_error * self.sigmoid(layer1,reverse=True)#*mod_learn_rate

        self.w1 += layer2.T.dot(layer2_delta)
        self.w0 += layer1.T.dot(layer1_delta)
        return np.mean(abs(layer2_error))
    def calcout(self,inp):
        inp=np.array(inp)
        layer0=inp
        layer1=self.sigmoid(np.dot(layer0,self.w0))
        out=self.sigmoid(np.dot(layer1,self.w1))
        return out

And the script importing said module:

import random
from numpy import mean
random.seed(50404)
op=open('Mall_Customers_Mod.txt','r')
full=op.read()
op.close()
full_lines=full.split('\n')
training_lines=random.sample(full_lines,175)
training_inputs=[]
training_outputs=[]
for j in training_lines:
    training_inputs.append([float(j.split(',')[0]),float(j.split(',')[1]),float(j.split(',')[2])])
    training_outputs.append(float(j.split(',')[3]))
testing_lines=random.sample(full_lines,175)
testing_inputs=[]
testing_outputs=[]
for l in testing_lines:
    testing_inputs.append([float(l.split(',')[0]),float(l.split(',')[1]),float(j.split(',')[2])])
    testing_outputs.append(float(l.split(',')[3]))
nn=hidden_net.network([3,9,1],seed=50404,learning_rate=10)
er=[]
txt=''
try:
    for i in range(10000):
        for l in range(len(training_inputs)):
            er.append(nn.train(training_inputs[l],training_outputs[l],10/(i+1)))
        if (i%1000==0 or i==1 or i==0 or i==2):
            print('epoch:{}\nerror:{}\nlearning_rate={}'.format(i,mean(er),10/(i+1)))
            txt=txt+'\nepoch:{}\nerror:{}'.format(i,mean(er))
            er=[]
except KeyboardInterrupt:
    pass
print('done!')
score=0
error=[]
tests=0
for i in range(len(testing_inputs)):
    print('net output: ' +str(nn.calcout(testing_inputs[i])))
    print('true output: '+str(testing_outputs[i]))
    error.append(abs(nn.calcout(testing_inputs[i]) - testing_outputs[i]))
print('error: {}'.format(mean(error)))
print('\n\nweights:{}'.format(nn.w0))

The normalized data has the form

Sex     Age          Income  Spending Score
0,0.019230769230769232,0.0,0.3877551020408163
0,0.057692307692307696,0.0,0.8163265306122449
1,0.038461538461538464,0.00819672131147541,0.05102040816326531
1,0.09615384615384616,0.00819672131147541,0.7755102040816326
1,0.25,0.01639344262295082,0.3979591836734694

I would've expected the output to vary, but it doesn't

net output: [0.49777196]
true output: 0.3979591836734694
net output: [0.49999907]
true output: 0.8571428571428571
net output: [0.49997918]
true output: 0.7346938775510204
net output: [0.49983941]
true output: 0.30612244897959184
net output: [0.49999985]

This seems like a pretty common error with neural networks, with many causes. I would add a bounty but i can't afford one. Being trying different things for ages, I hope one of you can figure it out! Thanks in advance, 3NiGMa

2
  • Hi. May I ask what is the meaning of "returning same" here? Do you mean the printed "net output" on different lines? Commented Apr 17, 2019 at 1:06
  • Yes, maybe I should’ve made that more clear Commented Apr 17, 2019 at 1:22

1 Answer 1

2

Your learning rate is too high for the size of your dataset, try upping it to 0.1 or 1 so your weights can change more rapidly

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.