How to build stacked Sequence-to-sequence autoencoder?

Question

in keras blog:"Building Autoencoders in Keras" the following code is provided to build single sequence to sequence autoencoder

from keras.layers import Input, LSTM, RepeatVector
from keras.models import Model

inputs = Input(shape=(timesteps, input_dim))
encoded = LSTM(latent_dim)(inputs)

decoded = RepeatVector(timesteps)(encoded)
decoded = LSTM(input_dim, return_sequences=True)(decoded)

sequence_autoencoder = Model(inputs, decoded)
encoder = Model(inputs, encoded)

I want to build stacked autoencoder, how to update this code to build stacked autoencoder?

i try it myself, and this is my code:

timesteps = 3
input_dim = 1
inputs = Input(shape=(timesteps, input_dim))
encoded = LSTM(4)(inputs)
encoded = RepeatVector(timesteps)(encoded)
encoded = LSTM(2)(encoded)
encoded = RepeatVector(timesteps)(encoded)
decoded = LSTM(4,return_sequences = True)(encoded)
decoded = LSTM(input_dim,return_sequences = True)(decoded)

sequence_autoencoder = Model(inputs, decoded)

sequence_autoencoder.compile(loss='mean_squared_error', optimizer='Adam')

sequence_autoencoder.fit(x_train, x_train,
epochs=100,
batch_size=1,
shuffle=True,
)

i want to know, is this code correct or am i missing something?

Daniel Möller · Accepted Answer · 2018-01-17 13:46:18Z

1

The code you created tries to follow the same follows the same philosophy that the example you found. But you're destroying the sequence too soon (and you ended up using an extra RepeatVector because of that).

To avoid this, you can use return_sequences=True in all encoder layers, except for the last. This keeps the sequence as a sequence allowing a greater interpretation power because you're not collapsing your data too soon.

#add return_sequences=True to all layers except for the last
encoded = LSTM(4, return_sequences=True)(inputs) 

#do not use RepeatVector, you've got your sequences preserved with their length

#the last encoder layer is the only one that collapses the sequence
encoded = LSTM(2)(encoded) 

#this RepeatVector is the only that is needed, to restore the sequence length
decoded = RepeatVector(timesteps)(encoded)

#the rest is the same
decoded = LSTM(4,return_sequences = True)(encoded)
decoded = LSTM(input_dim,return_sequences = True)(decoded)

answered Jan 17, 2018 at 13:46

Daniel Möller

86.8k24 gold badges202 silver badges222 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Mostafa Kotb Over a year ago

Another thing, the RepeatVector layer is a part of the encoder or the decoder. You add it with the decoder, isn't it suppose to be with the encoder?

Daniel Möller Over a year ago

No, the encoded data is the most condensed data you have.

Mostafa Kotb Over a year ago

Sorry Daniel, when i made the RepeatVector layer with the decoder, and trained SAE, then the output of the encoder became two dimension (batch,units) so, i can't feed it to another LSTM, it needs three dim.

Daniel Möller Over a year ago

You're free to do anything you want.

Collectives™ on Stack Overflow

How to build stacked Sequence-to-sequence autoencoder?

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related