You probably want this:

Nest[#^2 + b &, x, 2] // Expand

(* b + b^2 + 2 b x^2 + x^4 *)

Firstly note the order of arguments; function, initial argument for the function, and nesting depth. Secondly, Not the construction used for function; you have to make it clear what is the argument; it's marked by # in Mathematica, and & ends the Function. This is equivalent to more conventional form, where you name your function f and its' argument x:

f[x_] := x^2 + b

Nest[f, x, 2] // Expand

(* b + b^2 + 2 b x^2 + x^4 *)

Every convention above has a specific meaning; often you can write something that looks almost the same, but results are quite different. Eagerly head for the help center and tutorials; they're really good.

The last part in the above puzzle is the // Expand, which expands positive integer producs in the top level of the result of Nest. Without it, you get a mathematically identical result, but in a different form:

Nest[#^2 + b &, x, 2]

(* b + (b + x^2)^2 *)

Just to check, you can actually nest the function "manually", and this also results the symbolic result, assuming you haven't defined x earlier in the session:

f[x_] := x^2 + b

f[f[x]]

(* b + (b + x^2)^2 *)