You must first have a look at the very basics of Mathematica syntax: capitalization of commands and square brackets for the arguments of functions. Done that, your integral seems very complicated because of the three symbolic parameters. You can do special cases:
With[{a = 1, b = 0},
Integrate[(r^2 - 2 r (a Cos[\[Theta]] - b Sin[\[Theta]]) +
a^2 + b^2)^-1,
{r, 0, R}, {\[Theta], 0, \[Pi]},
Assumptions -> 0 < R < Sqrt[a^2 + b^2]]]
With[{a = 1, b = 0},
Integrate[(r^2 - 2 r (a Cos[\[Theta]] - b Sin[\[Theta]]) +
a^2 + b^2)^-1,
{r, 0, Sqrt[a^2 + b^2]}, {\[Theta], 0, \[Pi]}]]