Hi Sjoerd! This is not exactly about the post, but related to the Dropout method used in training evaluation mode. I am using the Self-normalizing Neural Network (SNN) for regression in one of my works. I had the idea to create a sample similar to the way you have generated here. I am not applying regularization for now. My questions are:

1. If I do not use regularization, can I just use the sample variance as the measure of the variance of my output? If not, could you explain the sentence

   The prior l=2 seems to work reasonably well, though in real applications you'd need to calibrate it with a validation set

in your post?

1. As SNNs use "AlphaDropout"s, (and I am using the default probability set in the one from Wolfram repo.) is this okay to do?

Yesterday I found the following paper (Dropout Inference in Bayesian Neural Networks with Alpha-divergences) by Yingzhen Li and Yarin Gal in which they address one of the shortcomings of the approach presented in the blog post. Basically, the method I showed above is based on Variational Bayesian Inference, which has a tendency to under fit the posterior (meaning that it gives more optimistic results than it should). To address this, they propose a modified loss function to train your neural network on.

In the attached notebook I tried to implement their loss function. It took a bit of tinkering, but I think this should work adequately. Other than that, I haven't given much thought yet to the calibration of the network and training parameters, which is definitely an important thing to do.

edit For those of who who're interested in understanding what the alpha parameter does in the modified loss function, it might be instructive to look at figure 2 in the following paper (Black-Box ?-Divergence Minimization) by Hernández-Lobato et al.,

Attachments:

I think it depends on how you do your GP regression in this case and what kernel you use. If you make a point estimate for the covariance length scale, I think you end up with constant prediction variance far from your data (since basically none of the data points are correlated with the prediction point). If you do a proper Bayesian inference where you integrate over the posterior distribution of your length scale, it will be different since you'll have contributions from very long length scales.

In the blog post by Yarin he also shows an example of GP regression with a square exponential where the error bands become constant.