Group Abstract

Message Boards

WOLFRAM COMMUNITY

6.3K Views

7 Replies

5 Total Likes

View groups...

Follow this post

Share this post:

GROUPS:

Tips for out-of-core training of neural net for semantic segmentation

Dan Farmer

Posted 3 years ago

POSTED BY: Dan Farmer

7 Replies

Sort By:

Kyle Jiang

Posted 3 years ago

Thanks! your solution do solve my problem with data generator function when `NetTrain`, that is: Image`ImportExportDump`DeleteCacheJPEG[]`

POSTED BY: Kyle Jiang

Dan Farmer

Posted 3 years ago

With the DeleteCachePNG tip I was able to get it working. If you want to play with it in my silly notebook above you can add (/ change the NetTrain call): LoadTrainingPair[idx_] := Module[{out}, out = <\|"Input" -> First@Image`ImportExportDump`ImageReadPNG[frogImages[[idx]]], "Output" -> First@Image`ImportExportDump`ImageReadPNG[gtImages[[idx]]]\|>; Image`ImportExportDump`DeleteCachePNG[]; out ] trainingGenerator = Function[ ParallelMap[LoadTrainingPair, RandomSample[Range[Length[frogImages]], #BatchSize]] ] trained = NetTrain[net, trainingGenerator, BatchSize -> 32, MaxTrainingRounds -> 100, TargetDevice -> "GPU"] That (and a variation on it for my work setup that was a bit different) allowed me to train without memory issues and still at 130-180 samples / sec. (the real version was tested on Ubuntu 20.04 + Mathematica 12.3.1 and the silly notebook above was tested on Windows 11 + Mathematica 13)

With the DeleteCachePNG tip I was able to get it working. If you want to play with it in my silly notebook above you can add (/ change the NetTrain call):

LoadTrainingPair[idx_] := Module[{out},
  out = <|"Input" -> 
     First@Image`ImportExportDump`ImageReadPNG[frogImages[[idx]]], 
    "Output" -> 
     First@Image`ImportExportDump`ImageReadPNG[gtImages[[idx]]]|>;
  Image`ImportExportDump`DeleteCachePNG[];
  out
  ]
trainingGenerator = Function[
  ParallelMap[LoadTrainingPair, 
   RandomSample[Range[Length[frogImages]], #BatchSize]]
  ]
trained = 
 NetTrain[net, trainingGenerator, BatchSize -> 32, 
  MaxTrainingRounds -> 100, TargetDevice -> "GPU"]

That (and a variation on it for my work setup that was a bit different) allowed me to train without memory issues and still at 130-180 samples / sec. (the real version was tested on Ubuntu 20.04 + Mathematica 12.3.1 and the silly notebook above was tested on Windows 11 + Mathematica 13)

POSTED BY: Dan Farmer

Kyle Jiang

Posted 3 years ago

POSTED BY: Kyle Jiang

Dan Farmer

Posted 3 years ago

Here's a simple example. The network itself is really dumb, don't pay any attention to that I was just trying to slap something together that would do some plausible amount of work on the GPU. On my machine (Windows 11, Mathematica 13 with Neural Net paclet version 13.0.3) this will eat all of the memory on my computer. If you elect to download the data I mentioned it's ~200 MB. Obviously that would fit in RAM if you just read it all at once, but for the actual case I'm dealing with that isn't an option.

POSTED BY: Dan Farmer

Giulio Alessandrini

Giulio Alessandrini, Wolfram Research Inc.

Posted 3 years ago

Assuming your segmentation masks are images, there was a memory leak in the fast image import function used but the NN framework which should have been fixed for 13.0.1 so be on the lookout for that update. In the meantime here are few things you can try: If the segmentations masks are small and you have less then 257 classes, save them in-core as byte arrays, they will take less space than 64bit integer arrays as they automatically gets save as Integer8 (mask = RandomInteger[10, {128, 128}]) // ByteCount BinarySerialize[mask] // ByteCount (* 131280 17512 ) Keep them out of core but export them as WXF files ( In[117] ) file1 = BinaryWrite["mask1", BinarySerialize[mask]]; Close["mask1"]; ( In[125] ) (mask1 = BinaryDeserialize@ReadByteArray[file1]) // MaxMemoryUsed // RepeatedTiming mask1 == mask ( Out[125] ) ( {0.000401338, 167864} ) ( Out[126] ) ( True ) Not sure why the one above is not using the same encoding but you can force it to use one byte per class ( In[122] ) file2 = BinaryWrite["mask2", BinarySerialize[NumericArray[mask, "Integer8"]]]; Close["mask2"]; ( In[128] ) (mask2 = BinaryDeserialize@ReadByteArray[file2]) // MaxMemoryUsed // RepeatedTiming ( Out[128] ) ( {0.00037028, 110752} ) ( In[130] ) mask2 == mask ( Out[130] ) ( True *) These solution would require the appropriate encoder on the segmentation port, e.g. ElementwiseLayer[Sin, "Input" -> NetEncoder[{"Function", BinaryDeserialize, {128, 128}}]]

Assuming your segmentation masks are images, there was a memory leak in the fast image import function used but the NN framework which should have been fixed for 13.0.1 so be on the lookout for that update.

In the meantime here are few things you can try:

If the segmentations masks are small and you have less then 257 classes, save them in-core as byte arrays, they will take less space than 64bit integer arrays as they automatically gets save as Integer8
```
(mask = RandomInteger[10, {128, 128}]) // ByteCount
BinarySerialize[mask] // ByteCount
(*
131280
17512
*)
```

Keep them out of core but export them as WXF files

(* In[117] *)
file1 = BinaryWrite["mask1", BinarySerialize[mask]];
Close["mask1"];

(* In[125] *)
(mask1 = BinaryDeserialize@ReadByteArray[file1]) // MaxMemoryUsed // RepeatedTiming
mask1 == mask

(* Out[125] *)
(* {0.000401338, 167864} *)

(* Out[126] *)
(* True *)

Not sure why the one above is not using the same encoding but you can force it to use one byte per class

(* In[122] *)
file2 = BinaryWrite["mask2", BinarySerialize[NumericArray[mask, "Integer8"]]];
Close["mask2"];

(* In[128] *)
(mask2 = BinaryDeserialize@ReadByteArray[file2]) // MaxMemoryUsed // RepeatedTiming

(* Out[128] *)
(* {0.00037028, 110752} *)

(* In[130] *)
mask2 == mask

(* Out[130] *)
(* True *)

These solution would require the appropriate encoder on the segmentation port, e.g.

ElementwiseLayer[Sin, "Input" -> NetEncoder[{"Function", BinaryDeserialize, {128, 128}}]]

POSTED BY: Giulio Alessandrini

Dan Farmer

Posted 3 years ago

POSTED BY: Dan Farmer

Giulio Alessandrini

Giulio Alessandrini, Wolfram Research Inc.

Posted 3 years ago

POSTED BY: Giulio Alessandrini

Reply to this discussion

Reply Preview

Attachments

Remove Add a file to this post

Follow this discussion

or Discard

Feedback