Message Boards Message Boards

GROUPS:

Avoid error training networks with 3D data?

Posted 2 months ago
313 Views
|
2 Replies
|
1 Total Likes
|

I am training some data using my onw neural net (UNET) which has ran great in version 11.

Using version 12 some updates have been done to the NetTrain function which are giving me troubles.

During training i get these errors.

CompiledFunction::cfta: Argument {Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]} at position 1 should be a rank 1 tensor of machine-size real numbers.

Transpose::nmtx: The first two levels of {{0.142857},If[Min[\[Infinity],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]]>0,Log10[{Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]}],{Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]}]} cannot be transposed.

CompiledFunction::cfta: Argument {Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]} at position 1 should be a rank 1 tensor of machine-size real numbers.

Transpose::nmtx: The first two levels of {{0.142857,0.285714},If[Min[\[Infinity],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]]>0,Log10[{Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]}],{Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]}]} cannot be transposed.

CompiledFunction::cfta: Argument {Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]} at position 1 should be a rank 1 tensor of machine-size real numbers.

General::stop: Further output of CompiledFunction::cfta will be suppressed during this calculation.

Transpose::nmtx: The first two levels of {{0.142857,0.285714,0.428571},If[Min[\[Infinity],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]]>0,Log10[{Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]}],{Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]}]} cannot be transposed.

General::stop: Further output of Transpose::nmtx will be suppressed during this calculation.

enter image description here

The errors only seem to happen when i uses large data. When training the netwrork with 2D data (128x128) i dont get any errors but when moving to 3D (40x128x128) the errors start popping up. The NetTrain still seems to run through its iterations but does not do anything.

Any experts around that know what is going on and how to fix it. Version 12 has some new features that i want to give a try so it would be a shame going back to version 11.

Thanks!

2 Replies

I kept changing parameters of NetTrain and it seams that the error is caused by GPU memory issues. If i use BatchSize->Automatic it works well if the images are less than 300pix x 300pix (8bit) in size. Im running the program in a GeForce 940MX, only has 2GB of memory (driver version 430.86). However it also works in larger images if I set the BatchSize to a lower number instead of leaving it in automatic (I used BatchSize->2 for images of 412x412). Let me know if this works for you. Im curious to know if this issue only occurs in windows or if it has to do with my specific GPU and GPU driver version.

I am having the exact same errors. I implemented a smaller version of U-Net in Mathematica 11.2 and it used to work perfectly in that version. Now that I upgraded to version 12, I get the error LibraryFunctionError[LIBRARYFUNCTIONERROR,6]. It only seams to occur if I use NetTrain in images that are bigger than 300pix x 300pix and if I am using TargetDevice->"GPU". Im working on Windows 10.

Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract