Hi everybody,
I would like to implement a custom norm regularisation on the weights of a LinearLayer. But Let's suppose for ease of discussion that I want to implement a L1 regularisation. In this case my objective will be to minimise a loss which is: (loss of task) + (Total[Abs[weights]])
I tried to play around with NetArray with without much success, here's an example:
net = NetGraph[
Association[
"linear" -> LinearLayer[1, "Weights" -> NetArray["c"]],
"reg" ->
FunctionLayer[{Total[
Abs[NetArray[<|"Name" -> "c", "Dimensions" -> 100|>]]]} &],
"thread" -> ThreadingLayer[#1 + #2 &]
],
{
NetPort["Input"] -> "linear",
"reg" -> "thread",
NetPort["linear", "Output"] -> "thread"
}]
That can be trained for example with:
dataTrain = Table[RandomReal[1, 100] -> {RandomReal[]}, 100];
trained = NetTrain[net, dataTrain]
But when I go to inspect the actual value of the weights, I get different results. That is:
NetExtract[trained, {"linear", "Weights"}]
is different from
NetExtract[trained, {"reg", "Net", 1, "Array"}]
How can I implement it? Do you have any ideas/comment/observations?
(I'm pretty sure that the example I gave is wrong as I would like to minimise the loss of the ouput of linear layer + the sum of the absolute value of the weights, but I think that the essence is the same)