The problem might be the singularity at the lower integration boundary. When x equals a the denominators become zero. You can also see that when you plot the integrand for some values a and b:

Plot[(1/Sqrt[(x - 1) (x^2 - a^2)]) - (1/Sqrt[(x + 1) (x^2 - a^2)]) /. a -> 1, {x, 1, 2}] 

You can also check that with Wolfram alpha:

WolframAlpha["Domain of (1/Sqrt[(x - 1) (x^2 - a^2)]) - (1/Sqrt[(x + \ 1) (x^2 - a^2)])", {{"Result", 1}, "Output"}]

This gives:

HoldComplete[(a < -1 && (-Sqrt[a^2] < x < -1 || x > Sqrt[a^2])) || (-1 <= a <= 1 && x > 1) || (a > 1 && (-Sqrt[a^2] < x < -1 || x > Sqrt[a^2]))]

When you solve the indefinite integral:

Integrate[(1/Sqrt[(x - 1) (x^2 - a^2)]) - 1/Sqrt[(x + 1) (x^2 - a^2)], x]

you obtain

(2 I Sqrt[1 - (-1 + a)/(-1 + x)] Sqrt[1 + (1 + a)/(-1 + x)] (-1 + x)^(3/2) Sqrt[-a^2 + x^2] EllipticF[I ArcSinh[Sqrt[1 + a]/Sqrt[-1 + x]](1 - a)/(1 + a)])/(Sqrt[1 + a] Sqrt[1 - a^2 + 2 (-1 + x) + (-1 + x)^2]Sqrt[(-1 + x) (-a^2 + x^2)]) - (2 I (1 + x)^(3/2) Sqrt[-a^2 + x^2] Sqrt[1 + (-1 + a)/(1 + x)] Sqrt[1 - (1 + a)/(1 + x)] EllipticF[I ArcSinh[Sqrt[-1 - a]/Sqrt[1 + x]], (1 - a)/(1 + a)])/(Sqrt[-1 - a] Sqrt[(1 + x) (-a^2 + x^2)] Sqrt[1 - a^2 - 2 (1 + x) + (1 + x)^2])

which is also clearly not defined at x==1.

Cheers, Marco

Even if you use Assumptions->1<a<b I believe Integrate will have trouble on attempting to find Limit of the antiderivative at the endpoints. I do eventually get a result for this.

Integrate[1/Sqrt[(x - 1)*(x^2 - a^2)] - 1/Sqrt[(x + 1)*(x^2 - a^2)], 
    {x, a, b}, Assumptions -> b > a > 1]

(2*((2*I)*Sqrt[(-1 + a^2)*(-1 + b)] - (2*I)*Sqrt[(-1 + a^2)*(1 + b)] + 
   Sqrt[(1 + a)*(-1 + b^2)]*EllipticF[ArcSin[Sqrt[(a + b)/a]/Sqrt[2]], 
     (2*a)/(-1 + a)] - Sqrt[(-1 + a)*(-1 + b^2)]*
    EllipticF[ArcSin[Sqrt[(a + b)/a]/Sqrt[2]], (2*a)/(1 + a)] - 
   Sqrt[(1 + a)*(-1 + b^2)]*EllipticK[(2*a)/(-1 + a)] + 
   Sqrt[(-1 + a)*(-1 + b^2)]*EllipticK[(2*a)/(1 + a)]))/
 Sqrt[(-1 + a^2)*(-1 + b^2)]