You must first have a look at the very basics of Mathematica syntax: capitalization of commands and square brackets for the arguments of functions. Done that, your integral seems very complicated because of the three symbolic parameters. You can do special cases:

With[{a = 1, b = 0},
 Integrate[(r^2 - 2 r (a Cos[\[Theta]] - b Sin[\[Theta]]) +
     a^2 + b^2)^-1,
  {r, 0, R}, {\[Theta], 0, \[Pi]},
  Assumptions -> 0 < R < Sqrt[a^2 + b^2]]]
With[{a = 1, b = 0},
 Integrate[(r^2 - 2 r (a Cos[\[Theta]] - b Sin[\[Theta]]) +
     a^2 + b^2)^-1,
  {r, 0, Sqrt[a^2 + b^2]}, {\[Theta], 0, \[Pi]}]]