This looks like another case of default branch cut selection producing unintuitive behavior. Branch cuts are a major source of problems and confusion when doing computer algebra and calculus. If you look on this and other similar forums, you'll find a plethora of cases where branch cut handling has caused confusion.
Clear[f, g];
f[m_][k_] = (m - h)^2/(k^2 + m^2 - m*h) /. h -> Sqrt[k^2 + m^2];
g[m_][k_] = FullSimplify[f[m][k]];
"f" and "g" are still equal to each other (generically) as you would expect:
Reduce[f[m][k] == g[m][k]]
True
Solving the limit doesn't require using Limit itself. You can do the following:
FullSimplify[f[m][0], Assumptions -> Element[m, Reals]]
FullSimplify[g[m][0], Assumptions -> Element[m, Reals]]
These both essentially give the following, good answer:
1 - Sign[m]
Which you can verify experimentally:
Manipulate[
Plot[{f[m][k], g[m][k]}, {k, -1, 1}, PlotRange -> {0, 2}], {m, -5, 5}]
So what is Limit doing?
The following isn't exactly what limit is doing, but it does demonstrate the fundamental issue with the choice of branch cuts. Let's look at the limit over the Numerator and Demoninator of f:
Limit[Numerator[f[m][k]], k -> 0]
2 m (m - Sqrt[m^2])
Limit[Denominator[f[m]], k -> 0]
m (m - Sqrt[m^2])
If you cancel m(m-Sqrt[m^2]) in both the numerator and denominator, you get 2. Mathematica
will do that cancelation:
2 m (m - Sqrt[m^2])/ (m (m - Sqrt[m^2]))
2
You could argue that's not a safe cancelation to do. Sqrt[m^2] is really either +m or -m and it could be either, but Mathematica assumes that they both mean essentially the same thing. It's not that m was ever assumed to be positive, it's the confusion above with Sqrt having 2 possible values. Of course the algebra gets done as though it has one value.
If you'd like, you can forward the example to support@wolfram.com. They can file it as an example for the developers to look at when they look into better ways to handle these kinds of symbolic issues.
How to avoid this problem
In this case, you can use
Surd instead of Sqrt:
Clear[f, g];
f[m_][k_] = (m - h)^2/(k^2 + m^2 - m*h) /. h -> Surd[k^2 + m^2,2];
g[m_][k_] = FullSimplify[f[m][k]];
Unfortunately, this isn't a solution to all possible branch cut problems. There isn't a clean way to handle them I'm afraid.
