Memory leak in neural network functionality?

Questions
, , ,
1

Hello, has anyone figured out how to actually delete a neural network from GPU memory without restarting wolfram desktop? There appears to be a pretty severe memory leak associated with using Neural Network functionality in WL.

For example,

$HistoryLength = 0
MemoryInUse\[\]
(* 68,363,752 *)

net =  NetModel[
  "Multi-scale Context Aggregation Net Trained on Cityscapes Data"]
mask = net[ImageResize[CurrentImage\[\], {128, 128}], 
  TargetDevice -> "GPU"]
MemoryInUse\[\]
(* 746866704 *)

ClearAll[mask, net]
ClearSystemCache\[\]
MemoryInUse\[\]
(* 746566120 *)

gpumem.PNG
gpumem.PNG743×353 11.2 KB

2

Using NeuralNetworks`ClearCache[] appears to free some CPU
memory, but it does nothing to the GB of data sitting in my valuable GPU memory.

This becomes a major problem when you are doing training and quickly trying to test architectures and restart the training process:

gpusad.PNG

wolframsad.PNG
wolframsad.PNG1196×371 9.26 KB

3

This is weird:

NeuralNetworks`MemoryUsageInfoString[]
"S:16,992M K:805M B:10 GPU1:558M GPU2:557M"

NeuralNetworks`MemoryUsageInfo[]
<|"System" -> 16987279360, "Kernel" -> 805609552, "Buckets" -> 10, 
 "GPU1" -> -6441992192, "GPU2" -> -6442254336|>