Sooo, I managed to implement a workaround by thresholding the IOU values before comparing them in my loss function (this might cause the model's confidence prediction distribution to be shaped incorrectly).
I am still pretty lost as to why directly using IOU values in the calculation directly causes "parameters to diverge"...
Maybe this information would help someone more knowledgeable about the back-end and what that error message is implying.
