Stefan -- That makes sense, although I've noticed that once it runs out of memory with a failure message, it seems like even a simpler network (that would succeed if I ran it first thing) can fail and crash the kernel, so there may also be an issue for how the kernel recovers state & resources after an out of memory condition (in addition to the problem you mentioned where it's internal checks on memory requirements fail).

PS The 970 (and possibly 4GB 960s) do have that weird Nvidia memory issue, where 4GB isn't really available. I have no idea whether that might confuse an application, but it is something memory-related that's unique to these cards. (Nvidia is having to send $30 to every 970 owner as a result -- not the 960, since Nvidia never configured it with 4GB, those were done by board vendors aftermarket).

I need to check with the developers, but what I think is happening is that the internal checks that guard against exhausting GPU memory during training are failing. Normally it should immediately see that training wouldn't work and bail out (giving the Failure["GPU memory exhausted"] answer), but here it fails to detect this and proceeds with training. Then when the GPU runs out of memory in the middle of the computations, the kernel doesn't know what to do and crashes.

What you describe sounds like a kernel crash. To be clear, when you run into the issue does the In/Out cell counter reset? I.e. if you immediately evaluated 1+1 it would look like

In[1]:= 1+1
Out[1]= 2

When I run the example on my machine (which has a GeForce 960 GTX) it returns with a Failure answer, saying that GPU memory was exhausted. Perhaps that is also happening on your machine but instead of properly bailing and returning a Failure the kernel is crashing...

Is there a way to access your training data? I.e. the files that are in "c:\\projects\\Submissions\\Thumbnails\\Selects", referenced in the notebook.