The basic problem is that the mesh refinement function needs to be compiled (because it is being used as a LibraryLink callback function). When something goes wrong with this automatic compilation, the discretization will proceed but the refinement function will be ignored.
I realize this is not made clear by the documentation, and in any case, there should be at least a warning message. Thank you for bringing this to our attention.
For the first example, try
ClearAll[meshRefine]
meshRefine[vertices_, area_] := area > 0.001
mesh = Polygon[{{1, 0}, {0, 1.8}, {-1, 0}}];
DiscretizeRegion[mesh, MeshRefinementFunction -> Function[{v, a}, Evaluate[meshRefine[v, a]]]]
For the last example, the compiler has trouble figuring out whether your mean is a rank-1 tensor or a scalar. Try
ClearAll[meshRefine]
meshRefine =
Function[{vertices, area},
Module[{meanv, meann}, meanv = Mean[vertices];
meann = Norm[meanv];
area > 0.001]];
mesh = Polygon[{{1, 0}, {0, 1.8}, {-1, 0}}];
DiscretizeRegion[mesh, MeshRefinementFunction -> meshRefine]