Group Abstract Group Abstract

Message Boards Message Boards

0
|
20.1K Views
|
8 Replies
|
6 Total Likes
View groups...
Share
Share this post:

Custom activation function in Neural Network?

Posted 8 years ago

Hello! Is it possible to set user defined activation function for a layer in neural network?

If this feature is not yet part of WL, are there any plans for such an addition as it would be very helpful.

POSTED BY: narendra
8 Replies
Posted 7 years ago

Here's an example how to implement somewhat more complex like differential equations: ODEs, SDEs, etc. into ElementwiseLayer[...]: DiffEq in Neural Net

(Open picture into new tab to see it in full size!)

Depending on the result representation with or without Normal[...] to get the value(s).

Attachments:
POSTED BY: C. H.

@narendra Replace x with # (Slot) which represent a pure function.

ElementwiseLayer[#*LogisticSigmoid[#] &]
POSTED BY: Alexey Golyshev
POSTED BY: Kapio Letto
Posted 8 years ago
POSTED BY: narendra
Posted 8 years ago

@Sebastian Bodenstein Yes, I am currently writing a paper on activation functions based on fractional calculus. I am currently implementing it in python, however Wolfram would be much better fit for it due to it having inbuilt functions from fractional calculus from which I derive my specific function.

However when I try to use it with ElementwiseLayer, it gives an error of ElementwiseLayer::invscf: my_func could not be symbolically evaluated as a unary scalar function.

(Interestingly, it is a unary scalar function because it can be plotted using only one variable and gives out scalar values when individually applied over scalar values)

POSTED BY: narendra

@narendra it is always helpful to post a complete code resulting in the error: http://wolfr.am/READ-1ST

POSTED BY: EDITORIAL BOARD

@narendra : out of curiosity, is there are particular activation you are trying to implement yourself?

Posted 8 years ago

ElementwiseLayer[f] allows you to define a layer with a custom activation function f applied to each element of the input:

http://reference.wolfram.com/language/ref/ElementwiseLayer.html.

POSTED BY: Yihe Dong
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard