PyTorch: initializing weight with numpy array + create a constant tensor

Question

I have the following code :

self.wi = nn.Embedding(num_embeddings, embedding_dim)
self.wj = nn.Embedding(num_embeddings1, embedding_dim)
self.bi = nn.Embedding(num_embeddings, 1)
self.bj = nn.Embedding(num_embeddings1, 1)

self.wi.weight.data.uniform_(-1, 1)
self.wj.weight.data.uniform_(-1, 1)
self.bi.weight.data.zero_()
self.bj.weight.data.zero_()

I want to initialize the weights with numpy array, and I want to create a constant tensor, which is also a numpy array. I am new to PyTorch, and I appreciate any help.

If I understand you correctly, you might want to split up your question into separate parts (initializing with the numpy.array, and then also a different question about the constant tensor). Otherwise, you can generally look for answers on how to initialize embeddings with other values, see here — dennlinger
– dennlinger, Commented Dec 12, 2019 at 10:14

dennlinger · Accepted Answer · 2019-12-12 10:21:33Z

You can initialize embedding layers with the function nn.Embedding.from_pretrained().

In your specific case, you would still have to firstly convert the numpy.array to a torch.Tensor, but otherwise it is very straightforward:

import torch as t
import torch.nn as nn
import numpy as np

# This can be whatever initialization you want to have
init_array = np.zeros([num_embeddings, embedding_dims])

# As @Daniel Marchand mentioned in his answer, 
# you do have to cast it explicitly as a tensor, otherwise it won't work.
wi = nn.Embedding.from_pretrained(t.tensor(init_array), freeze=False)

The parameter freeze=False is important if you still want to train your network afterwards, as otherwise you would keep the embeddings at the same constant values. Generally, .from_pretrained is used to "transfer" learned embeddings, but it does work for your case, too.

Daniel Marchand · Accepted Answer · 2019-12-11 21:28:13Z

0

You may wish to look at converting a numpy array into a torch tensor How to convert Pytorch autograd.Variable to Numpy?

answered Dec 11, 2019 at 21:28

Daniel Marchand

6441 gold badge10 silver badges31 bronze badges

2 Comments

Abrar Over a year ago

I need to initialize the tensor not convert it

Daniel Marchand Over a year ago

I don't think it should matter, see pytorch.org/tutorials/beginner/nn_tutorial.html

Collectives™ on Stack Overflow

PyTorch: initializing weight with numpy array + create a constant tensor

2 Answers 2

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related