Message Boards Message Boards

Custom Activation Function in Neural Network?

GROUPS:

Hello! Is it possible to set user defined activation function for a layer in neural network?

If this feature is not yet part of WL, are there any plans for such an addition as it would be very helpful.

POSTED BY: narendra
Answer
11 days ago

ElementwiseLayer[f] allows you to define a layer with a custom activation function f applied to each element of the input:

http://reference.wolfram.com/language/ref/ElementwiseLayer.html.

POSTED BY: Yihe Dong
Answer
11 days ago

@narendra : out of curiosity, is there are particular activation you are trying to implement yourself?

Answer
9 days ago

@Sebastian Bodenstein Yes, I am currently writing a paper on activation functions based on fractional calculus. I am currently implementing it in python, however Wolfram would be much better fit for it due to it having inbuilt functions from fractional calculus from which I derive my specific function.

However when I try to use it with ElementwiseLayer, it gives an error of ElementwiseLayer::invscf: my_func could not be symbolically evaluated as a unary scalar function.

(Interestingly, it is a unary scalar function because it can be plotted using only one variable and gives out scalar values when individually applied over scalar values)

POSTED BY: narendra
Answer
9 days ago

@narendra it is always helpful to post a complete code resulting in the error: http://wolfr.am/READ-1ST

POSTED BY: Moderation Team
Answer
5 days ago

@S├ębastien Guillet While I can not disclose the exact details as I am currently not finished writing the paper yet, ElementWiseLayer does not do well with many functions beyond the normal commonly used ones.

Example: The recently popular Swish Activation Function which performs better than ReLU in most cases. It is defined as x*sigmoid(x)

elem = ElementwiseLayer[x*LogisticSigmoid[x]]

gives an error

ElementwiseLayer::invscf: x LogisticSigmoid[x] could not be symbolically evaluated as a unary scalar function.

This is the same error I am receiving whenever the complexity of function is a little more than exponential or tanh.

POSTED BY: narendra
Answer
1 day ago

Group Abstract Group Abstract