It seems that Solve needs a little help: the equation becomes easy by expressing Cos[2 \[CurlyPhi]] in terms of Cos[ \[CurlyPhi]]:
eq = \[Lambda] == (-d Cos[\[CurlyPhi]] +
Sqrt[(d^2 Cos[2 \[CurlyPhi]] + 2 r^2 - d^2)/2])/
m (Sqrt[(\[Epsilon] nd^2)/(\[Epsilon] - nd^2)] - ni Sin[\[Alpha]]);
expandedEq =
eq /. Cos[2 \[CurlyPhi]] :> TrigExpand[Cos[2 \[CurlyPhi]]];
FullSimplify[Solve[expandedEq, \[CurlyPhi]]]
It is disappointing that TrigExpand does not expand the Cos in simple cases like this:
TrigExpand[Sqrt[Cos[2 x]]]
