David P. -- I'm running on Windows 10. I did a NetTrain with TargetDevice -> GPU and it ran just fine on my EVGA 1080. Just to double check after seeing your post, I ran the code again, and indeed the GPU lights up with activity, so it is really in use.

Just a comment to Christopher's answer.

There is an efficient workflow for transfer learning in general that is worth mentioning: instead of keeping the pre-trained part of the net active during training, use that to extract features from the whole dataset, dump them to disk and train the final part on top of the dumped features. This will make training experiments much faster at the price of extra disk space being used. Since you usually run many trainings, exploring different combinations of hyperparameters and net architectures, the time gain is usually significant.

Following Christopher's example, you woud do:

extractor = Take[net, 7];
features = extractor[trainingSet[[All, 1]]];
featureTrainingSet = Thread[features -> trainingSet[[All, 2]]];
Export["feature_training_set.mx", featureTrainingSet, "MX"];

Where the above assumes that trainingSet is a list of the form {input1 -> label1, input2 -> label2, ...}.

When ready for training just load the new dataset, then define the new layers and train them:

featureTrainingSet = Import["feature_training_set.mx"];
newLayers = NetChain[{...}];
trained = NetTrain[newLayers, featureTrainingSet]

After training, you can obtain your final model by simply doing

final = NetChain[{extractor, trained}]

Just to mention: Drop could also be used (maybe more similar to pop). Both Take and Drop work on NetChain, whilst Take works on NetGraph as well. This is mentioned in the NetChain and NetGraph docs.

Hi David.

I think what you are looking for is LearningRateMultipliers. LearningRateMultipliers is an option for NetTrain that lets you specify what learning rate you want for each layer in the network.

Look at the details section for more information, but these seem to be the most important lines:

"m;;n->r use multiplier r for layers m through n"

"If r is zero or None, it specifies that the layer or array should not undergo training and will be left unchanged by NetTrain."

"LearningRateMultipliers->{layer->None} can be used to "freeze" a specific layer."

Here is an example. First, we will get some model from NetModel:

net = NetModel["LeNet Trained on MNIST Data"]

Then we can get some training and test data:

resource = ResourceObject["MNIST"];
trainingData = ResourceData[resource, "TrainingData"];
testData = ResourceData[resource, "TestData"];

And finally, we can train just the layers after layer 7:

NetTrain[net, trainingData, ValidationSet -> testData, LearningRateMultipliers -> {1 ;; 7 -> None}]

This particular example probably isn't all too useful, as we're probably using the same dataset as this network was originally trained on, but I think this shows the general idea.