I am not sure what is causing the error messages, but it is most likely related to a value you previously defined that you are using within Integrate in a way that violates the function's syntax. Please try restarting Mathematica to begin with.
The first step is to make sure you are using Mathematica's syntax. In Mathematica, built-in functions are capitalized and wrap their arguments in square brackets. So the integral can be written out as:
Integrate[(1)/(Sqrt[1 - x^2 - y^2])*
Exp[(-(a - x)^2 - (b - y)^2)*(mu^2/2)], {x, -Infinity,
Infinity}, {y, -Infinity, Infinity}]
This integral appears to be difficult to solve for generic values of a, b, and mu. You can use the Assuming expression to give Mathematica assumptions about the values of those parameters. For example, we could assume that they are Real numbers:
Assuming[Element[{a, b, mu}, Reals],
Integrate[(1)/(Sqrt[1 - x^2 - y^2])*
Exp[(-(a - x)^2 - (b - y)^2)*(mu^2/2)], {x, -Infinity,
Infinity}, {y, -Infinity, Infinity}]
]
Mathematica takes a while with this expression and hasn't produced a solution for me yet. It is possible that it may not find a symbolic solution or that a symbolic solution doesn't exist. If that is the case, then this problem has to be evaluted numerically.
