This is weird:
NeuralNetworks`MemoryUsageInfoString[] "S:16,992M K:805M B:10 GPU1:558M GPU2:557M" NeuralNetworks`MemoryUsageInfo[] <|"System" -> 16987279360, "Kernel" -> 805609552, "Buckets" -> 10, "GPU1" -> -6441992192, "GPU2" -> -6442254336|>
Using NeuralNetworks`ClearCache[] appears to free some CPU memory, but it does nothing to the GB of data sitting in my valuable GPU memory.
This becomes a major problem when you are doing training and quickly trying to test architectures and restart the training process:
