# Avoid error training networks with 3D data?

Posted 2 months ago
313 Views
|
2 Replies
|
1 Total Likes
|
 I am training some data using my onw neural net (UNET) which has ran great in version 11.Using version 12 some updates have been done to the NetTrain function which are giving me troubles. During training i get these errors. CompiledFunction::cfta: Argument {Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]} at position 1 should be a rank 1 tensor of machine-size real numbers. Transpose::nmtx: The first two levels of {{0.142857},If[Min[\[Infinity],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]]>0,Log10[{Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]}],{Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]}]} cannot be transposed. CompiledFunction::cfta: Argument {Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]} at position 1 should be a rank 1 tensor of machine-size real numbers. Transpose::nmtx: The first two levels of {{0.142857,0.285714},If[Min[\[Infinity],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]]>0,Log10[{Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]}],{Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]}]} cannot be transposed. CompiledFunction::cfta: Argument {Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]} at position 1 should be a rank 1 tensor of machine-size real numbers. General::stop: Further output of CompiledFunction::cfta will be suppressed during this calculation. Transpose::nmtx: The first two levels of {{0.142857,0.285714,0.428571},If[Min[\[Infinity],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]]>0,Log10[{Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]}],{Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]],Mean[LibraryFunctionError[LIBRARY_FUNCTION_ERROR,6]]}]} cannot be transposed. General::stop: Further output of Transpose::nmtx will be suppressed during this calculation. The errors only seem to happen when i uses large data. When training the netwrork with 2D data (128x128) i dont get any errors but when moving to 3D (40x128x128) the errors start popping up. The NetTrain still seems to run through its iterations but does not do anything.Any experts around that know what is going on and how to fix it. Version 12 has some new features that i want to give a try so it would be a shame going back to version 11.Thanks!
2 Replies
Sort By:
Posted 1 month ago
 I kept changing parameters of NetTrain and it seams that the error is caused by GPU memory issues. If i use BatchSize->Automatic it works well if the images are less than 300pix x 300pix (8bit) in size. Im running the program in a GeForce 940MX, only has 2GB of memory (driver version 430.86). However it also works in larger images if I set the BatchSize to a lower number instead of leaving it in automatic (I used BatchSize->2 for images of 412x412). Let me know if this works for you. Im curious to know if this issue only occurs in windows or if it has to do with my specific GPU and GPU driver version.