Message Boards Message Boards


[WSS18] Introducing Hadamard Binary Neural Networks

Posted 1 year ago
2 Replies
4 Total Likes

Introducing Hadamard Binary Neural Networks

Deep neural networks are an important tool in modern applications. It has become a major challenge to accelerate their training. As the complexity of our training tasks increase, the computation does too. For sustainable machine learning at scale, we need distributed systems that can leverage the available hardware effectively. This research hopes to exceed the current state of the art performance of neural networks by introducing a new architecture optimized for distributability. The scope of this work is not just limited to optimizing neural network training for large servers, but also to bring training to heterogeneous environments; paving way for a distributed peer to peer mesh computing platform that can harness the wasted resources of idle computers in a workplace for AI.

Network Architecture and Layer Evaluator

Here, I will describe the network and the Layer Evaluator, to get an in depth understanding of the network architecture.


  • hbActForward : Forward binarization of Activations.

  • hbWForward : Forward binarization of Weights.

  • binAggression : Aggressiveness of binarization (Vector length to binarize)

Set up the Layer Evaluator.

layerEval[x_, layer_Association] := layerEval[x, Lookup[layer, "LayerType"], Lookup[layer, "Parameters"]];
layerEval[x_, "Sigmoid", param_] := 1/(1 + Exp[-x]);
layerEval[x_, "Ramp", param_] := Abs[x]*UnitStep[x];
layerEval[ x_, "LinearLayer", param_] := Dot[x, param["Weights"]];
layerEval[ x_, "BinLayer", param_] := Dot[hbActForward[x, binAggression], hbWForward[param["Weights"], binAggression]];
layerEval[x_, "BinarizeLayer", param_] := hbActForward[x, binAggression];
netEvaluate[net_, x_, "Training"] := FoldList[layerEval, x, net];
netEvaluate[net_, x_, "Test"] := Fold[layerEval, x, net];

Define the network

net = {<|"LayerType" -> "LinearLayer", "Parameters" -> <|"Weights" -> w0|>|>,
          <|"LayerType" -> "Ramp"|>,
          <|"LayerType" -> "BinarizeLayer"|>,
          <|"LayerType" -> "BinLayer", "Parameters" -> <|"Weights" -> w1|>|>, 
          <|"LayerType" -> "Ramp"|>,
          <|"LayerType" -> "BinLayer", "Parameters" -> <|"Weights" -> w2|>|>, 
          <|"LayerType" -> "Sigmoid"|>       };

MatrixForm@netEvaluate[net, input[[1 ;; 3]], "Test" ] (* Giving network inputs *)

enter image description here

Advantages of Hadamard Binarization

  • Faster convergence with respect to vanilla binarization techniques.
  • Consistently about 10 times faster than CMMA algorithm.
  • Angle of randomly initialized vectors preserved in high dimensional spaces. (Approximately 37 degrees as vector length approach infinity.)
  • Reduced communication times for distributed deep learning.
  • Optimization of im2col algorithm for faster inference.
  • Reduction of model sizes.

Accuracy analysis

enter image description here

As seen above, the HBNN model gives 87% accuracy, whereas the BNN model (Binary Neural Networks) give only 82%. These networks have only been trained for 5 epochs.

Performance Analysis

X Axis: Matrix Size | Y Axis: Time (seconds)


enter image description here


$\hspace{1mm}$enter image description here

Visualize weight histograms

enter image description here

It is evident that the Hadamard BNN preserves the distribution of the weights much better. Note that the BNN graph has a logarithmic vertical axis, for representation purposes.

Demonstration of the angle preservation ability of the HBNN architecture

enter image description here

Binarization approximately preserves the direction of high dimensional vectors. The figure above demonstrates that the angle between a random vector (from a standard normal distribution) and its binarized version converges to ~ 37 degrees as the dimension of the vector goes to infinity. This angle is exceedingly small in high dimensions.

2 Replies

enter image description here - Congratulations! This post is now a Staff Pick as distinguished by a badge on your profile! Thank you, keep it coming!

hbActForward ? hbWForward ? binAggression ?

w0, w1 ,w2 ?

Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
or Discard

Group Abstract Group Abstract