how to Solve this eq in mathematica
f[f[a+b]]==f[2a]+2f[b]
I am afraid there is no general rule to cope with problems of this kind.
Although mathematica is an extraordinary powerful tool I am afraid that will not work. You have to use your brain.
f[x_] := 2 x In[93]:= f[f[a + b]] // Simplify Out[93]= 4 (a + b) In[94]:= f[2 a] + 2 f[b] // Simplify Out[94]= 4 (a + b)