I kept changing parameters of NetTrain and it seams that the error is caused by GPU memory issues. If i use BatchSize->Automatic it works well if the images are less than 300pix x 300pix (8bit) in size. Im running the program in a GeForce 940MX, only has 2GB of memory (driver version 430.86). However it also works in larger images if I set the BatchSize to a lower number instead of leaving it in automatic (I used BatchSize->2 for images of 412x412). Let me know if this works for you. Im curious to know if this issue only occurs in windows or if it has to do with my specific GPU and GPU driver version.