Training a dense layer along with an lstm layer

I'm trying to set my initial hidden states in the lstm network using a dense layer but, I have been having problems with using multiple inputs in my network. The flow of the network looks something like below. The dense layer between static features (input) and init_hidden_states should be trainable. So far, I tried using a DAG network but, failed due to an error (If there is sequential input layer, there can't be any other input layer present). In my case the sequential input layer is used for time series input and I'm using an image input layer to input the static features. Any advice or example is appreciated. Thank you.

Answer 1

[Asvin Kumar]

I suppose what you're asking for is to have an LSTM layer in which the initial values of hidden state is trainable. If that's the case, I'm afraid this is not currently possible. The initial value of hidden state of an LSTM is a property of the layer, the 'HiddenState' property, and it is a state parameter which is not learnable.

Refer the following doc page and the linked example for more details: HiddenState - LSTM Layer and LSTM Layer and Properties.

[Yildirim Kocoglu]

Thank you for letting me know Asvin. Yes, that’s what I meant: for initial values of the hidden state to be trainable. I was also looking to find out how the dimensions of the initial hidden state is decided: if I have num_hidden_units=50 in lstm, I end up having initial hidden state with dimensions (50,1). I was wondering where this “1” is from? When does it change to “2” for instance (if it does)? Thank you again.