This is very interesting. As someone who is not intimately familiar with training NNs I have a question however:

You mentioned one of the applications here was training a sufficiently deep NN to compress a photo library. Let's say you have a NN that has already been trained on a library; what if I wanted to add one more photo to the library? Would I have to retrain the NN from scratch and do a large number of training steps, or is the NN with its current parameters a good starting point, and we can retrain the current NN with a small number of training steps?

I'm guessing this type of question is something that is far more general, but it is something I was wondering about.