Group Abstract Group Abstract

Message Boards Message Boards

0
|
1.2K Views
|
0 Replies
|
0 Total Likes
View groups...
Share
Share this post:

Implementing normalization across groups of channels

Posted 2 years ago

GroupNormalization is as important as LayerNormalization and BatchNormalization. The current NormalizationLayer doesn't have an option for normalizing across groups of channels. It can be implemented by a sequence of reshaping and normalization layers.

GroupNormalization[nGroups_, inputShape_] := 
   NetChain[{ 
(*assuming the channel dimension is the first one*)
    ReshapeLayer[{nGroups, inputShape[[1]]/nGroups, Inherited, 
      Automatic}], NormalizationLayer[2 ;;, "Complement"], 
    ReshapeLayer[inputShape]}, "Input" -> inputShape]

It would be great if the number of groups option was added to LayerNormalization or create a new layer for only this purpose.

Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard