I would do it this way:
f[x_] := (x + 3)^2; f[f[x]]
Beware the difference between square brackets and round parentheses, and between the pattern x_ and the variable x.
x_
x
Thanks!
//Ulf