Keras LSTM autoencoder with embedding layer

Question

I am trying to build a text LSTM autoencoder in Keras. I want to use an embedding layer but I'am not sure how to implement this. The code looks like this.

inputs = Input(shape=(timesteps, input_dim))
embedding_layer = Embedding(numfeats + 1,
                            EMBEDDING_DIM,
                            weights=[data_gen.get_embedding_matrix()],
                            input_length=maxlen,
                            trainable=False)

embedded_sequence = embedding_layer(inputs)
encoded = LSTM(num_units)(inputs)

decoded = RepeatVector(timesteps)(encoded)
decoded = LSTM(???, return_sequences=True)(decoded)

sequence_autoencoder = Model(inputs, decoded)

sequence_autoencoder.compile(loss='binary_crossentropy', optimizer='adam')

I am not sure how to decode the output into the target sequence (which is obviously the input sequence).

Did you figure out how to do this?

sariii
– sariii

2019-04-06 23:21:59 +00:00
Commented Apr 6, 2019 at 23:21 — sariii
– sariii, Commented Apr 6, 2019 at 23:21

sebdei · Accepted Answer · 2019-11-03 18:07:03Z

1

You can first convert the word to embeddings and pass them to the fit()

expected_output = np.array([[embedding_matrix[word_index] for word_index in encoded_sequence] for encoded_sequence in padded_sequences])
history = lstm_autoencoder.fit(padded_sequences, expected_output, epochs=15, verbose=1)

answered Nov 3, 2019 at 18:07

sebdei

111 bronze badge

Sign up to request clarification or add additional context in comments.

1 Comment

Alex_Gidiotis Over a year ago

My understanding is that this is an example of the first solution suggested by @ChrisYao. The expected output is the embedding of the input sequence and the output layer is a dense layer with shape (n_words, embedding_dim). The only issue is that it can be hard optimising this objective.

Chris Yao · Accepted Answer · 2018-01-11 21:29:26Z

0

There is no way to implement an inversed embedding layer in the decoder, because the embedding layer is not differentiable. There are probably other ways to work around:

construct the autoencoder from the output of the embedding layer, to a layer with a similar dimension. Then use the nearest neighbor or other algorithms to generate the word sequence from there.
construct an asymmetric autoencoder, using the time distributed layer and dense layers to reduce the dimension of LSTM output.

Hopefully this helps.

answered Jan 11, 2018 at 21:29

Chris Yao

2403 silver badges13 bronze badges

2 Comments

sariii Over a year ago

Can you please reference an example?

Alex_Gidiotis Over a year ago

The first solution can work without needing the mapping into word tokens. The expected output is the embedding of the input sequence and the output layer is a dense layer with shape (n_words, embedding_dim). The only issue is that it can be hard optimising this objective.

Collectives™ on Stack Overflow

Keras LSTM autoencoder with embedding layer

2 Answers 2

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related