One approach is to make a parametrized guess and then use Reduce. For example, try

g[c0_,c1_,c2_][x_]:=c0+c1*x+c2*x^2

and then use Reduce on a small set of examples.

FullSimplify[Reduce[ Flatten[Table[f[f[a+b]]==f[2a]+2f[b],{a,0,5},{b,0,5}]]/.f->g[c0,c1,c2], {c0,c1,c2}]]
(* c2 == 0 && ((c0 == 0 && c1 == 0) || c1 == 2) *)

I added FullSimplify so it would get rid of redundant results which can be confusing.

It is telling us that either f is identically zero or f[x]==2x+constant

Note: if this is a homework question for a math class, it should be noted that there is some subtlety with the domain and other complications arise if f is not continuous. In general, use the powerful functions like Reduce and Solve, but you are also supposed to exper

Thanks for HELP I'm thinking is there a general rule TO SOLVE This kind of equation for example f[f[x]]==f[x] __________ f[f[f[x]]]==g[f[x]]

ReSolve is a good tool put Cant solve a nest case ill try to make a small code to solve it put first i need general rule

How can I know f[x]=2x by any kind of mathematica tool for example LIKE Rsolve ,