Pytorch nn.Linear different output for same input

Question

For learning purposes, I'm trying to build a simple perceptron with pytorch which should not be trained but just give the output for set weights. Here's the code:

import torch.nn
from torch import tensor

class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = torch.nn.Linear(3,1)
        self.relu = torch.nn.ReLU()
        # force weights to equal one
        with torch.no_grad():
            self.fc1.weight = torch.nn.Parameter(torch.ones_like(self.fc1.weight))

    def forward(self, x):
        x = self.fc1(x)
        output = self.relu(x)
        return output

net = Net()
test_tensor = tensor([1, 1, 1])
print(net(test_tensor.float()).item())

I expect this single layer neural network to output 3. And that is roughly(!) the output for every execution, but it ranges from 2.5 to 3.5. Where does randomness enter the model?

Berriel · Accepted Answer · 2021-06-25 21:26:28Z

2

Q: Where does randomness enter the model?

It comes from the bias init. As you can see here, the bias is not initialized to zero as you expected.

You can fix it this way:

import torch
from torch import nn

class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = torch.nn.Linear(3,1)
        self.relu = torch.nn.ReLU()
        # force weights to equal one
        with torch.no_grad():
            torch.nn.init.ones_(self.fc1.weight)
            torch.nn.init.zeros_(self.fc1.bias)

    def forward(self, x):
        x = self.fc1(x)
        output = self.relu(x)
        return output

x = torch.tensor([1., 1., 1.])
Net()(x)
# >>> tensor([3.], grad_fn=<ReluBackward0>)

answered Jun 25, 2021 at 21:26

Berriel

13.8k4 gold badges51 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pytorch nn.Linear different output for same input

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related