Group Abstract Group Abstract

Message Boards Message Boards

Sequence-to-sequence regression with recurrent neural networks

20 Replies
Posted 4 years ago

Hi Sjoerd the changes you gave fixed the issue, thank you (although I haven't tried training it yet). I'll try it out with version 11.3 when that's available.

POSTED BY: Steve Walker
Posted 4 years ago

Hi Sjoerd I looked at the NetFoldOperator version of your seq2seq regression but couldn't get it to evaluate: decoderNetInternal function fails complaining about CatenateLayer "Type inconsistency in CatenateLayer: \!(\"specified level (2) cannot exceed rank of lowest-rank input (Min[1, Var[0]])\")." Might this be a version related issue? I'm running version 12.2.

POSTED BY: Steve Walker

Hello Steve. Yes, it seems there was a small change in how types are inferred in the network since V12.2. You should be able to fix the issue by changing:

{"rec", "attention"} ->  "catenate" -> "lin" -> NetPort["Prediction"]

to

{NetPort["rec", "Output"], "attention"} ->  "catenate" -> "lin" -> NetPort["Prediction"]

I updated the notebook accordingly. I haven't tested the rest of the code in 12.2, so thanks for bringing this to my attention!

POSTED BY: Sjoerd Smit

Thanks for your quick answer! I will try later and hope it will work (I correct exams right now...)

Hi Christos. Glad to hear it's working. As for using other financial data: without further details I can only recommend that you first try to mold your data into a matrix with the same shape as trainingData in my example. That is: a matrix with the time-axis (regularly sampled) as the first dimension and the different indices (or whatever you're trying to predict) in the 2nd dimension. After that, you should be able to generate inputs for the network with the line

trainingExamples = genFun[trainingData, {10, 6, 1}][<|"BatchSize" -> 3|>]

Next, verify that the loss network (i.e., the one that goes into NetTrain) initializes correctly with NetInitialize and returns real-valued losses when you apply it to the trainingExamples you just generated.

POSTED BY: Sjoerd Smit
Posted 6 years ago
POSTED BY: jerome ibanes

One method that I want to try and implement once my V12 installation works properly again, is to change the way the network is trained. Right now, the output sequence is fed into the Prediction port with a delay to simulate the predictions the network will make. However, with the changes to NetFoldOperator (in particular the ability to feed constant vectors into these layers), it should be possible to train the network on its own predictions, making the training process more similar to the way it is used afterwards. That's what I want to try out next.

POSTED BY: Sjoerd Smit

The most recent paclet update to the neural networks fixed a bug in NetFoldOperator that stopped me before from being able to use it for this application. I updated the main post with a new notebook, so please take a look if you're interested.

POSTED BY: Sjoerd Smit
Posted 6 years ago
POSTED BY: jerome ibanes
Posted 6 years ago

i am also stuck at the same thing looking forward for solution.

POSTED BY: ANTONY OTTO

What are you stuck on? Have you checked the notebook I attached to the original post this morning? That code should work in V12.

POSTED BY: Sjoerd Smit
Posted 6 years ago

Thank you, looking forward to this.

POSTED BY: jerome ibanes

I'm actually struggling with V12 myself, so apologies if it takes a bit longer than expected ;).

POSTED BY: Sjoerd Smit

@jerome: I just updated the main post with a notebook with the V12 version of the code.

POSTED BY: Sjoerd Smit
Posted 6 years ago

Sjoerd, could you update it for Mathematica 12 and provide a link to a notebook?

Thanks in advance.

POSTED BY: jerome ibanes

Sure, I'd be happy to :). I can't guarantee that I can do that right away, but be sure to check back in a few days.

POSTED BY: Sjoerd Smit
Posted 4 years ago

Dear Sjoerd,

May I have your email address? I have a question about the LSTM model.

Thank you so much.

POSTED BY: Alex Teymouri

Hi Alex. Please check my profile for my LinkedIn account. You can send me a personal message there and maybe set up a Zoom call. I prefer not to post my e-mail address publicly; I hope you understand. I'm also a bit busy at the moment, so please forgive me if I don't respond too quickly.

POSTED BY: Sjoerd Smit

enter image description here - Congratulations! This post is now a Staff Pick as distinguished by a badge on your profile! Thank you, keep it coming, and consider contributing your work to the The Notebook Archive!

POSTED BY: EDITORIAL BOARD
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard