Message Boards Message Boards

0
|
6623 Views
|
4 Replies
|
3 Total Likes
View groups...
Share
Share this post:

Cloning layers in neural network ?

Lucien Grondin, hobbyist
Posted 4 years ago

Is there a way to specify that two layers in a neural network should be identical (that is, they must share the exact same weights and biases) ?

I've tried to duplicate a trivial linear layer in a net chain and clearly during training both version get their own weights and biases. Is there a way to avoid this instantiation ?

FYI, I'm looking into this with the goal of implementing this work.
POSTED BY: Lucien Grondin
4 Replies
Sort By:
Alec Graves
Alec Graves
Posted 4 years ago

I was curious about this myself, and it turns out there is a way to easily share weights between layers of a network.

https://reference.wolfram.com/language/ref/NetInsertSharedArrays.html

layer = NetInsertSharedArrays[LinearLayer[1,"Input"->1]]
net = NetChain[{layer, layer}] // NetInitialize

It seems to work by automagically converting any arrays to NetSharedArray references that will be shared between all copies of the NetInsertSharedArray'd network (layer in my example).

That looks like a really cool project! I had not heard of Growing Neural Cellular Automata before. Best of luck!
POSTED BY: Alec Graves

Very nice and convenient function, thanks for pointing it out.

Turns out it works with arbitrary networks. I'll post the notebook of what I've done so far. Apparently applying this function to the single-step network functions out-of-the-box, even though that network is quite complicated.

I'll add the notebook of my current progress. (It's very early though).

PS. I've just learned about NetFoldOperator, which is possibly preferable to a long chain of nets with shared parameters.

Creating a net such as NetChain[ConstantArray[oneStepNet,100]] seemed to lag at least the interface, so it's possibly not a good idea. I have a hard time believing this would be trainable by the way, but that's what they did in the paper, so what do I know.

PS#2. Or even NetNestOperator. Damn, so many things to know.
POSTED BY: Lucien Grondin

Lucien-san,

I hope the following might be what you'll expect.

net2 = NetChain[{#, #} &[
    LinearLayer[1, "Weights" -> NetSharedArray["W"], 
      "Biases" -> NetSharedArray["B"], "Input" -> 1] // 
     NetInitialize]];

net2 = NetTrain[net2, 
   Table[{x} -> {.5 - .1 # + RandomReal[{-.01, .01}] &@x}, {x, 
     RandomReal[{-1, 1}, 10]}]];
Table[Normal@NetExtract[NetExtract[net2, layer], param], {layer, 
  2}, {param, {"Weights", "Biases"}}]
POSTED BY: Kotaro Okazaki

Yep. It's exactly what I wanted.

For some reason while looking at net-related functions I had searched the pattern *Layer instead of Net*, I guess that's why I had never seen this function before.

Thanks :)
POSTED BY: Lucien Grondin
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Tag limit exceeded
Note: Only the first five people you tag will receive an email notification; the other tagged names will appear as links to their profiles.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract