You should use NetReplacePart to change the input size:
NetReplacePart[NetTake[vgg, {NetPort["Input"], "relu4_3"}], "Input" -> {3, 64, 64}]
