Message Boards

GROUPS:

Harshal Gajjar

Creating a custom recurrent neural network layer

Harshal Gajjar, Georgia Institute of Technology

Posted 1 year ago

2820 Views

2 Replies

13 Total Likes

Follow this post

Creating a custom recurrent neural network layer
by Harshal Gajjar

In this post, we will implement the LongShortTermMemoryLayer from scratch. This will work to lay the foundations to implement any recurrent layer variants. We will implement LSTM in 4 parts, each one having an image to guide.

To concretize things, let’s assume that hidden state is a 32 dimensional vector and the input to LSTM (at every time step) is 128 dimensional.

In[]:=

inputDimension=128;hiddenStateDimension=32;

Implementing first part

These neat images were taken from Christopher Olah’s blog, anyways getting back to its implementation:

In[]:=

lstmCell=NetGraph[{(*1*)LinearLayer[hiddenStateDimension,"Input"inputDimension],(*2*)LinearLayer[hiddenStateDimension,"Input"hiddenStateDimension],(*3*)TotalLayer[],(*4*)ElementwiseLayer[LogisticSigmoid](*f_t*)},{NetPort["old_h"]23,NetPort["Input"]13,34(*f_t*)},"old_h"{hiddenStateDimension}]

Out[]=

NetGraph



uniniti

aliz

Number of inputs:	2
Output port:	vector (size: 32)



Implementing second part

In[]:=

Out[]=

NetGraph



uniniti

aliz

Number of inputs:	2
Number of outputs:	3



Implementing third part

In[]:=

lstmCell=NetGraph[{(*1*)LinearLayer[hiddenStateDimension,"Input"inputDimension],(*2*)LinearLayer[hiddenStateDimension,"Input"hiddenStateDimension],(*3*)TotalLayer[],(*4*)ElementwiseLayer[LogisticSigmoid](*f_t*),(*5*)LinearLayer[hiddenStateDimension,"Input"inputDimension],(*6*)LinearLayer[hiddenStateDimension,"Input"hiddenStateDimension],(*7*)TotalLayer[],(*8*)ElementwiseLayer[LogisticSigmoid](*i_t*),(*9*)LinearLayer[hiddenStateDimension,"Input"inputDimension],(*10*)LinearLayer[hiddenStateDimension,"Input"hiddenStateDimension],(*11*)TotalLayer[],(*12*)ElementwiseLayer[Tanh](*c_tilde_t*),(*13*)ThreadingLayer[Times],(*14*)ThreadingLayer[Times],(*15*)TotalLayer[](*new_c*)},{NetPort["Input"]13,NetPort["old_h"]23,34(*f_t*),NetPort["Input"]57,NetPort["old_h"]67,78(*i_t*),NetPort["Input"]911,NetPort["old_h"]1011,1112(*c_tilde_t*),413,NetPort["old_c"]13,814,1214,1315,1415(*new_c*),15NetPort["new_c"]},"old_h"{hiddenStateDimension},"old_c"{hiddenStateDimension}]

Out[]=

NetGraph



uniniti

aliz

Number of inputs:	3
new_c port:	vector (size: 32)



Implementing last part

There’s a slight difference here - I pass

through another LinearLayer and then pass that as the output (upward arrow).

In[]:=

lstmCell=NetGraph[{(*1*)LinearLayer[hiddenStateDimension,"Input"inputDimension],(*2*)LinearLayer[hiddenStateDimension,"Input"hiddenStateDimension],(*3*)TotalLayer[],(*4*)ElementwiseLayer[LogisticSigmoid](*f_t*),(*5*)LinearLayer[hiddenStateDimension,"Input"inputDimension],(*6*)LinearLayer[hiddenStateDimension,"Input"hiddenStateDimension],(*7*)TotalLayer[],(*8*)ElementwiseLayer[LogisticSigmoid](*i_t*),(*9*)LinearLayer[hiddenStateDimension,"Input"inputDimension],(*10*)LinearLayer[hiddenStateDimension,"Input"hiddenStateDimension],(*11*)TotalLayer[],(*12*)ElementwiseLayer[Tanh](*c_tilde_t*),(*13*)ThreadingLayer[Times],(*14*)ThreadingLayer[Times],(*15*)TotalLayer[](*new_c*),(*16*)LinearLayer[hiddenStateDimension,"Input"inputDimension],(*17*)LinearLayer[hiddenStateDimension,"Input"hiddenStateDimension],(*18*)TotalLayer[],(*19*)ElementwiseLayer[LogisticSigmoid](*o_t*),(*20*)ElementwiseLayer[Tanh],(*21*)ThreadingLayer[Times],(*22*)LinearLayer[hiddenStateDimension]},{NetPort["Input"]13,NetPort["old_h"]23,34(*f_t*),NetPort["Input"]57,NetPort["old_h"]67,78(*i_t*),NetPort["Input"]911,NetPort["old_h"]1011,1112(*c_tilde_t*),413,NetPort["old_c"]13,814,1214,1315,1415(*new_c*),15NetPort["new_c"],NetPort["Input"]1618,NetPort["old_h"]1718,1819(*o_t*),152021,1921(*new_h*),21NetPort["new_h"],2122NetPort["Output"]},"old_h"{hiddenStateDimension},"old_c"{hiddenStateDimension}]

Out[]=

NetGraph



uniniti

aliz

Number of inputs:	3
Number of outputs:	3



Wrapping into a function

Let’s wrap the above code into a function to generate new custom layers as and when needed:

lstmCell[hiddenStateDimension_,inputDimension_]:=NetGraph[{(*1*)LinearLayer[hiddenStateDimension,"Input"inputDimension],(*2*)LinearLayer[hiddenStateDimension,"Input"hiddenStateDimension],(*3*)TotalLayer[],(*4*)ElementwiseLayer[LogisticSigmoid](*f_t*),(*5*)LinearLayer[hiddenStateDimension,"Input"inputDimension],(*6*)LinearLayer[hiddenStateDimension,"Input"hiddenStateDimension],(*7*)TotalLayer[],(*8*)ElementwiseLayer[LogisticSigmoid](*i_t*),(*9*)LinearLayer[hiddenStateDimension,"Input"inputDimension],(*10*)LinearLayer[hiddenStateDimension,"Input"hiddenStateDimension],(*11*)TotalLayer[],(*12*)ElementwiseLayer[Tanh](*c_tilde_t*),(*13*)ThreadingLayer[Times],(*14*)ThreadingLayer[Times],(*15*)TotalLayer[](*new_c*),(*16*)LinearLayer[hiddenStateDimension,"Input"inputDimension],(*17*)LinearLayer[hiddenStateDimension,"Input"hiddenStateDimension],(*18*)TotalLayer[],(*19*)ElementwiseLayer[LogisticSigmoid](*o_t*),(*20*)ElementwiseLayer[Tanh],(*21*)ThreadingLayer[Times],(*22*)LinearLayer[hiddenStateDimension]},{NetPort["Input"]13,NetPort["old_h"]23,34(*f_t*),NetPort["Input"]57,NetPort["old_h"]67,78(*i_t*),NetPort["Input"]911,NetPort["old_h"]1011,1112(*c_tilde_t*),413,NetPort["old_c"]13,814,1214,1315,1415(*new_c*),15NetPort["new_c"],NetPort["Input"]1618,NetPort["old_h"]1718,1819(*o_t*),152021,1921(*new_h*),21NetPort["new_h"],2122NetPort["Output"]},"old_h"{hiddenStateDimension},"old_c"{hiddenStateDimension}]

Using NetFoldOperator to convert LSTM cell into a recurrent computation layer:

In[]:=

lstmLayer[hiddenStateDimension_,inputDimension_]:=NetFoldOperator[lstmCell[hiddenStateDimension,inputDimension],{"new_c""old_c","new_h""old_h"}]

An example

This one is from LongShortTermMemoryLayer documentation but reworked using custom LSTM layer. It takes in a string “1+10” and outputs the result of that computation.

In[]:=

enc=NetEncoder[{"Characters",{DigitCharacter,"+"}}];makeRule[a_,b_]:=IntegerString[a]<>"+"<>IntegerString[b]{a+b};trainingData=Flatten@Table[makeRule[i,j],{i,0,99},{j,0,99}];RandomSample[trainingData,4]//InputForm

Out[]//InputForm=

{60+47 -> {107}, 9+47 -> {56}, 34+64 -> {98}, 69+81 -> {150}}

Creating the custom layers:

In[]:=

lstmLayer1=lstmLayer[40,11]

Out[]=

NetFoldOperator



uniniti

alized

Input:	matrix (size: n ×11)
new_c:	matrix (size: n ×40)
new_h:	matrix (size: n ×40)
Output:	matrix (size: n ×40)



In[]:=

lstmLayer2=lstmLayer[20,40]

Out[]=

NetFoldOperator



uniniti

alized

Input:	matrix (size: n ×40)
new_c:	matrix (size: n ×20)
new_h:	matrix (size: n ×20)
Output:	matrix (size: n ×20)



In[]:=

net=NetInitialize@NetGraph[{UnitVectorLayer[],lstmLayer1,lstmLayer2,SequenceLastLayer[],LinearLayer[1]},{NetPort["Input"]12,NetPort[2,"Output"]3,NetPort[3,"Output"]45},"Input"enc,"Output""Scalar"]

Out[]=

NetGraph



InputPort
Input:	string
OutputPorts
new_c1:	matrix (size: n ×40)
new_h1:	matrix (size: n ×40)
new_c2:	matrix (size: n ×20)
new_h2:	matrix (size: n ×20)
Output:	scalar



Creating a training net:

In[]:=

lossNet=NetGraph[<|"net"net,"loss"MeanSquaredLossLayer[]|>,{{NetPort["net","Output"],NetPort["Target"]}"loss"NetPort["Loss"]}]

Out[]=

NetGraph



Number of inputs:	2
Number of outputs:	5



In[]:=

trainedLossNet=NetTrain[lossNet,trainingData,BatchSize64,MaxTrainingRounds200,ValidationSetScaled[0.1],LossFunction"Loss"]

Out[]=

NetGraph



Number of inputs:	2
Number of outputs:	5



Extracting trained net and adding net encoder:

In[]:=

net=NetExtract[trainedLossNet,"net"];net=NetChain[{net},"Input"enc]

Out[]=

NetChain



Input port:	string
Number of outputs:	5



Getting network results:

In[]:=

Round@net[{"5+10","10+25","9+11","44+44"}]["Output"]

Out[]=

{{15},{35},{20},{88}}

Voila, you have a custom RNN layer implemented!

References

◼

https://colah.github.io/posts/2015-08-Understanding-LSTMs/

◼

https://www.bioinf.jku.at/publications/older/2604.pdf

◼

https://en.wikipedia.org/wiki/Long_short-term_memory

POSTED BY: Harshal Gajjar

2 Replies

Sort By:

Moderation Team

Moderation Team, WOLFRAM

Posted 1 year ago

-- you have earned *Featured Contributor Badge* Your exceptional post has been selected for our editorial column *Staff Picks* http://wolfr.am/StaffPicks and Your Profile is now distinguished by a *Featured Contributor Badge* and is displayed on the Featured Contributor Board. Thank you!

POSTED BY: Moderation Team

Michel Mesedahl

Posted 8 months ago

I get this error on evaluation: SetDelayed::write: Tag NetGraph in NetGraph[<\|Type->Graph,<<3>>,Edges-> {NeuralNetworks`NetPath[Nodes,1,Inputs,Input]->NeuralNetworks`NetPath[Inputs,Input], <<31>>,NeuralNetworks`NetPath[Outputs,Output]- >NeuralNetworks`NetPath[Nodes,22,Outputs,Output]}\|>,<\|Version->12.3.3,Unstable->False\|>] [hiddenStateDimension_,inputDimension_] is Protected.

I get this error on evaluation:

SetDelayed::write: Tag NetGraph in NetGraph[<|Type->Graph,<<3>>,Edges->
{NeuralNetworks`NetPath[Nodes,1,Inputs,Input]->NeuralNetworks`NetPath[Inputs,Input],
<<31>>,NeuralNetworks`NetPath[Outputs,Output]-
>NeuralNetworks`NetPath[Nodes,22,Outputs,Output]}|>,<|Version->12.3.3,Unstable->False|>]   
[hiddenStateDimension_,inputDimension_] is Protected.

POSTED BY: Michel Mesedahl

Reply to this discussion

Reply Preview

Attachments

Remove Add a file to this post

Follow this discussion

or Discard

Group Abstract

Feedback