I've struggled with NotebookEvaluate[] as well when run from a subkernel. One of my workarounds for simpler notebooks is to write my own "NotebookEvaluate" function:

1) read the raw form of the notebook using Get

2) replace every Input cell type with a sequence of Input and Output cells, where the Output cell has gone from its BoxForm, ToExpression, and back to BoxForm.

evaluateNotebook[loc_String] := Module[{test,nb},
  UsingFrontEnd[
    test=Get[loc];
    nb=CreateDocument[test/.Cell[BoxData[x___],"Input",y___] :> Sequence[Cell[BoxData[x],"Input",y],Cell[BoxData[ToBoxes[ToExpression[BoxData[x]]]],"Output"]]];
    NotebookSave[nb,loc];
    NotebookClose[nb];
  ]
]

This will not overwrite older Output or warning messages, but you can add some rules to remove those, too, using empty sequences:

  (*rule to remove old output cells*)
  Cell[BoxData[___], "Output", ___] -> Sequence[]; 

  (*rules to remove old message cells*)
  Cell[BoxData[___], "Message", ___] -> Sequence[]; 

I'm not sure how robust this approach is, i.e. I haven't tested how it deals with cells that are commented out, but it does a good job for how short it is. I'm open to any feedback on improving it.

I need to correct what I said. Doing this causes hang only in version 9.0.1 and 10.0.2, but not in 10.3.1, 10.4.1 or 11.0.1.

However, there is a problem in recent versions too: the subsidiary kernel process does not terminate after closing the link. It hangs around indefinitely.

Let me give a concrete example:

Start a kernel:

link = LinkLaunch["\"" <> First[$CommandLine] <> "\" -mathlink"];
LinkRead[link]; (*read initial prompt*)

Send the evaluation:

LinkWrite[link,
 Unevaluated@EvaluatePacket[
   UsingFrontEnd[
    nb = CreateDocument[{Cell["1+1", "Input"]}];
    NotebookEvaluate[nb, InsertResults -> True];
    NotebookGet[nb]
    ]
   ]
 ]

In 10.0.2 at this point the subsidiary kernel stops responding. In v11 we can continue:

In[4]:= LinkRead[link]

Out[4]= ReturnPacket[
 Notebook[{Cell[
    CellGroupData[{Cell["1+1", "Input"], 
      Cell[OutputFormData["2"], "Output", 
       CellChangeTimes -> {3.6894*10^9}]}, Open]]}, 
  WindowSize -> {808, 751}, 
  WindowMargins -> {{316, Automatic}, {Automatic, 50}}, 
  FrontEndVersion -> 
   "11.0 for Mac OS X x86 (32-bit, 64-bit Kernel) (September 21, \
2016)", StyleDefinitions -> "Default.nb"]]

As you can see, the evaluation succeeded, and I got back a notebook expression.

Now let's close the link:

LinkClose[link]

At this point the link becomes invalid. However, the process on the other end of the link doesn't terminate. Neither does the front end it has launched. I must kill it using kill -9 as a simple kill won't do it.

Luckily there is a workaround. Instead of LinkClose, just send a Quit[]:

LinkWrite[link, Unevaluated@EvaluatePacket[Quit[]]]

And then close: LinkClose[link].

Question: Is there a way to make this work in version 10.0, even if it requires some hacks?

Why do I need this? I want to process some documentation notebooks precisely with a selected version of Mathematica. This is to avoid the "InsufficientVersionWarning" on the users' computers. What I really need is using a front end of a given version, not a kernel. So if there's a way to use a v10.0 kernel from v11.0 without going through a v10.0 kernel, that would be a workable solution for me.