You probably want this:
Nest[#^2 + b &, x, 2] // Expand
(* b + b^2 + 2 b x^2 + x^4 *)
Firstly note the order of arguments; function, initial argument for the function, and nesting depth. Secondly, Not the construction used for function; you have to make it clear what is the argument; it's marked by # in Mathematica, and & ends the Function. This is equivalent to more conventional form, where you name your function f and its' argument x:
f[x_] := x^2 + b
Nest[f, x, 2] // Expand
(* b + b^2 + 2 b x^2 + x^4 *)
Every convention above has a specific meaning; often you can write something that looks almost the same, but results are quite different. Eagerly head for the help center and tutorials; they're really good.
The last part in the above puzzle is the // Expand, which expands positive integer producs in the top level of the result of Nest. Without it, you get a mathematically identical result, but in a different form:
Nest[#^2 + b &, x, 2]
(* b + (b + x^2)^2 *)
Just to check, you can actually nest the function "manually", and this also results the symbolic result, assuming you haven't defined x earlier in the session:
f[x_] := x^2 + b
f[f[x]]
(* b + (b + x^2)^2 *)
