Message Boards Message Boards

0
|
5029 Views
|
1 Reply
|
0 Total Likes
View groups...
Share
Share this post:

Optimizing images export in binary format

Posted 9 years ago

Hi everyone. I am trying to generate a small images 20x20 with white symbols on black backround and export them in binary format for training my Neural Net. I need a bunch of images which are rotated and scaled on different values. In total I have about 4000 images to export.

(More details: I binarize images and save a bunch of zeros and ones all in one file. I also have a description file which contains lines describing symbol and its transformation)

Here is a code I use and I have problems with it. It is slow and it leads to "not enough memory". I completely do not understand this memory issue, I have about 3.5Gb free for it.

I appriciate any help from anyone, thank you very much.

dim = 20;

digit[\[Alpha]_, s_, d_] := 
 Binarize[Graphics[
   Text[Style[ToString[d], Black, s {dim, dim}], {0., 0.}, 
    Center, {Cos[\[Alpha]/360. 2 \[Pi]], Sin[\[Alpha] 2 \[Pi]/360.]}],
    AspectRatio -> 1, ImageSize -> {dim, dim}], {0, 0.5}]

DescriptionString[symb_, w_, h_, deg_, scale_] := 
 symb <> "\t" <> ToString[w] <> "\t" <> ToString[h] <> "\t" <> 
  ToString[deg] <> "\t" <> ToString[scale]

SaveDataSet[dir_, symbols_, w_, h_, degrees_, scales_] := Module[
   {images, descriptions, ostreamDesc, ostreamImag},
   DeleteDirectory[dir, DeleteContents -> True];
   CreateDirectory[dir];
   ostreamDesc = OpenWrite[dir <> "/dict"];
   ostreamImag = OpenWrite[dir <> "/images.bin", BinaryFormat -> True];
   Do[
    With[{scale = Random[Real, {0.4, 1.0}]},
     With[{img = ImageData@digit[degree, scale, symb], 
       desc = DescriptionString[symb, w, h, degree, scale]},
      BinaryWrite[ostreamImag, img]; WriteLine[ostreamDesc, desc];]],
    {symb, symbols}, {degree, degrees}];
   Close[ostreamDesc];
   Close[ostreamImag];
   ];

(*THIS LEADS TO OUT OF MEMORY ISSUE*)
SaveDataSet["e:/work/HL/Digits/BinDigitsRnd", {"0", "1", "2", "3", 
  "4","5","6","7","8","9"}, dim, dim, Range[0.0, 360.0 - 0.00001, 1.0], {0.9}]
Posted 9 years ago

Hi Alexander,

Which Mathematica version do you use? I have tried your code with version 10.3 and got a couple of errors. During the run, the memory used by the FrontEnd grows up to 400 Mb what I would consider as a memory leak similar to this one. But I don't get the "not enough memory" message on my laptop with 8 Gb of RAM running Windows 7 x64.

The Kernel <-> FrontEnd communication during rasterization of the Graphics expressions (which you perform by applying Binarize) is the bottleneck which results in your code being slow. You can speed up your code and avoid the memory leak if you Rasterize your digits not one-by-one (as you currently do) but for example 1000 digits at one time.

POSTED BY: Alexey Popkov
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract