Message Boards Message Boards

0
|
872 Views
|
0 Replies
|
0 Total Likes
View groups...
Share
Share this post:

Implementing normalization across groups of channels

Posted 1 year ago

GroupNormalization is as important as LayerNormalization and BatchNormalization. The current NormalizationLayer doesn't have an option for normalizing across groups of channels. It can be implemented by a sequence of reshaping and normalization layers.

GroupNormalization[nGroups_, inputShape_] := 
   NetChain[{ 
(*assuming the channel dimension is the first one*)
    ReshapeLayer[{nGroups, inputShape[[1]]/nGroups, Inherited, 
      Automatic}], NormalizationLayer[2 ;;, "Complement"], 
    ReshapeLayer[inputShape]}, "Input" -> inputShape]

It would be great if the number of groups option was added to LayerNormalization or create a new layer for only this purpose.

Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract