I let Mathematica run overnight the following code
Integrate[(r^2 Cos[A r Cos[x]] Exp[-r^2/2])/(a^2 Cos[x]^2 +
b^2 Sin[x]^2), {r, 0, Infinity}, {x, 0, Pi/2}]
but it was unable to compute the closed form in that time. However, Mathematica also didnt give up. Do you think Mathematica can do it?
I suspect that the Cos inside a Cos makes it impossible. It looks like an integral in polar coordinates. Might be easier in Cartesian coordinates.
Another thing to point out, is that there isn’t always an answer to an integral, as my teacher best put it “99% of all integrals don’t have antiderivatives.” So try making those constant coefficients into variables and write the integral as a function of 3 variables (assuming those three constants are A, a, and b) so that you can work around that instead. You can also plot the newly defined function with respect to two variables and change the third one each time you want to run it. That’s my best suggestion. If you want to know more, just ask.
Doing the r under assumptions is the first step
In[2]:= Integrate[(r^2 Cos[A r Cos[x]] Exp[-r^2/2])/(a^2 Cos[x]^2 +
b^2 Sin[x]^2), {r, 0, Infinity},(* {x,0,Pi/2} *)
Assumptions -> {A > 0, a > 0, b > 0}]
Out[2]= (E^(-(1/2) A^2 Cos[x]^2) Sqrt[\[Pi]/2] (1 - A^2 Cos[x]^2))/(
a^2 Cos[x]^2 + b^2 Sin[x]^2)
now look for a meaningful substitution, you need 0 \leq x \leq \frac{\pi}{2}. And as a preliminary step try a = b to get rid of the denominator.