For 11.0 and 11.1 there is only one way of defining a custom loss: build it out of existing net layers (any net that has a scalar output can be used for training). There are many more layers coming in 11.1 which will allow for a great deal more flexibility in defining your own loss functions (including the loss you are trying to define above).
We are looking into supporting users writing loss functions that cannot be written as compositions of existing layers, but this will only be ready earliest 11.2.
