Doing the r
under assumptions is the first step
In[2]:= Integrate[(r^2 Cos[A r Cos[x]] Exp[-r^2/2])/(a^2 Cos[x]^2 +
b^2 Sin[x]^2), {r, 0, Infinity},(* {x,0,Pi/2} *)
Assumptions -> {A > 0, a > 0, b > 0}]
Out[2]= (E^(-(1/2) A^2 Cos[x]^2) Sqrt[\[Pi]/2] (1 - A^2 Cos[x]^2))/(
a^2 Cos[x]^2 + b^2 Sin[x]^2)
now look for a meaningful substitution, you need
$0 \leq x \leq \frac{\pi}{2}$. And as a preliminary step try
$a = b$ to get rid of the denominator.