With a bit of cheating we can get candidate solutions for x^2 and y^2 as below. The caveat is that they might be incorrect due to branch cut issues.
exprs = {1/2 \[Lambda] y^(1/2)
x^(-(1/2)) (p - 1/2 x^2 - 1/2 y^2) + (a -
k p + \[Lambda] (x y)^(1/2)) (-x),
1/2 \[Lambda] x^(1/2)
y^(-(1/2)) (p - 1/2 x^2 - 1/2 y^2) + (a -
k p + \[Lambda] (x y)^(1/2)) (-y)};
e2 = Numerator[Together[exprs]];
e3 = PowerExpand[e2 /. {x -> x^2, y -> y^2}]
(* Out[11]= {-4 a x^3 + 4 k p x^3 + 2 p y \[Lambda] - 5 x^4 y \[Lambda] -
y^5 \[Lambda], -4 a y^3 + 4 k p y^3 + 2 p x \[Lambda] -
x^5 \[Lambda] - 5 x y^4 \[Lambda]} *)
Solving this modified system is not hard.
Timing[solnsb = Solve[e3 == 0, {x, y}];]
Out[21]= {1.356464, Null}
There are 25 solutions. I show the first three are shown below. One would need to take square roots though, since we solved (modulo branch cut issues) for (x^2, y^2) pairs rather than (x,y).
{{x -> 0,
y -> 0}, {x -> (
1/(-2 a p \[Lambda] +
2 k p^2 \[Lambda]))((
4 a^2 (3 p - (2 a^2)/\[Lambda]^2 + (4 a k p)/\[Lambda]^2 - (
2 k^2 p^2)/\[Lambda]^2 - (
2 Sqrt[(a - k p)^2 (a^2 - 2 a k p + k^2 p^2 -
3 p \[Lambda]^2)])/\[Lambda]^2)^(3/4))/(3 Sqrt[3]) - (
8 a k p (3 p - (2 a^2)/\[Lambda]^2 + (4 a k p)/\[Lambda]^2 - (
2 k^2 p^2)/\[Lambda]^2 - (
2 Sqrt[(a - k p)^2 (a^2 - 2 a k p + k^2 p^2 -
3 p \[Lambda]^2)])/\[Lambda]^2)^(3/4))/(3 Sqrt[3]) + (
4 k^2 p^2 (3 p - (2 a^2)/\[Lambda]^2 + (4 a k p)/\[Lambda]^2 - (
2 k^2 p^2)/\[Lambda]^2 - (
2 Sqrt[(a - k p)^2 (a^2 - 2 a k p + k^2 p^2 -
3 p \[Lambda]^2)])/\[Lambda]^2)^(3/4))/(3 Sqrt[3]) - (
p \[Lambda]^2 (3 p - (2 a^2)/\[Lambda]^2 + (
4 a k p)/\[Lambda]^2 - (2 k^2 p^2)/\[Lambda]^2 - (
2 Sqrt[(a - k p)^2 (a^2 - 2 a k p + k^2 p^2 -
3 p \[Lambda]^2)])/\[Lambda]^2)^(3/4))/Sqrt[
3] + (\[Lambda]^2 (3 p - (2 a^2)/\[Lambda]^2 + (
4 a k p)/\[Lambda]^2 - (2 k^2 p^2)/\[Lambda]^2 - ({{x -> 0,
y -> 0}, {x -> (
1/(-2 a p \[Lambda] +
2 k p^2 \[Lambda]))((
4 a^2 (3 p - (2 a^2)/\[Lambda]^2 + (4 a k p)/\[Lambda]^2 - (
2 k^2 p^2)/\[Lambda]^2 - (
2 Sqrt[(a - k p)^2 (a^2 - 2 a k p + k^2 p^2 -
3 p \[Lambda]^2)])/\[Lambda]^2)^(3/4))/(3 Sqrt[3]) - (
8 a k p (3 p - (2 a^2)/\[Lambda]^2 + (4 a k p)/\[Lambda]^2 - (
2 k^2 p^2)/\[Lambda]^2 - (
2 Sqrt[(a - k p)^2 (a^2 - 2 a k p + k^2 p^2 -
3 p \[Lambda]^2)])/\[Lambda]^2)^(3/4))/(3 Sqrt[3]) + (
4 k^2 p^2 (3 p - (2 a^2)/\[Lambda]^2 + (4 a k p)/\[Lambda]^2 - (
2 k^2 p^2)/\[Lambda]^2 - (
2 Sqrt[(a - k p)^2 (a^2 - 2 a k p + k^2 p^2 -
3 p \[Lambda]^2)])/\[Lambda]^2)^(3/4))/(3 Sqrt[3]) - (
p \[Lambda]^2 (3 p - (2 a^2)/\[Lambda]^2 + (
4 a k p)/\[Lambda]^2 - (2 k^2 p^2)/\[Lambda]^2 - (
2 Sqrt[(a - k p)^2 (a^2 - 2 a k p + k^2 p^2 -
3 p \[Lambda]^2)])/\[Lambda]^2)^(3/4))/Sqrt[
3] + (\[Lambda]^2 (3 p - (2 a^2)/\[Lambda]^2 + (
4 a k p)/\[Lambda]^2 - (2 k^2 p^2)/\[Lambda]^2 - (
2 Sqrt[(a - k p)^2 (a^2 - 2 a k p + k^2 p^2 -
3 p \[Lambda]^2)])/\[Lambda]^2)^(7/4))/(3 Sqrt[3])),
y -> -((3 p - (2 a^2)/\[Lambda]^2 + (4 a k p)/\[Lambda]^2 - (
2 k^2 p^2)/\[Lambda]^2 - (
2 Sqrt[(a - k p)^2 (a^2 - 2 a k p + k^2 p^2 -
3 p \[Lambda]^2)])/\[Lambda]^2)^(1/4)/Sqrt[3])},
{x -> (1/(-2 a p \[Lambda] +
2 k p^2 \[Lambda]))(-((
4 I a^2 (3 p - (2 a^2)/\[Lambda]^2 + (4 a k p)/\[Lambda]^2 - (
2 k^2 p^2)/\[Lambda]^2 - (
2 Sqrt[(a - k p)^2 (a^2 - 2 a k p + k^2 p^2 -
3 p \[Lambda]^2)])/\[Lambda]^2)^(3/4))/(3 Sqrt[3])) + (
8 I a k p (3 p - (2 a^2)/\[Lambda]^2 + (4 a k p)/\[Lambda]^2 - (
2 k^2 p^2)/\[Lambda]^2 - (
2 Sqrt[(a - k p)^2 (a^2 - 2 a k p + k^2 p^2 -
3 p \[Lambda]^2)])/\[Lambda]^2)^(3/4))/(3 Sqrt[3]) - (
4 I k^2 p^2 (3 p - (2 a^2)/\[Lambda]^2 + (4 a k p)/\[Lambda]^2 - (
2 k^2 p^2)/\[Lambda]^2 - (
2 Sqrt[(a - k p)^2 (a^2 - 2 a k p + k^2 p^2 -
3 p \[Lambda]^2)])/\[Lambda]^2)^(3/4))/(3 Sqrt[3]) + (
I p \[Lambda]^2 (3 p - (2 a^2)/\[Lambda]^2 + (
4 a k p)/\[Lambda]^2 - (2 k^2 p^2)/\[Lambda]^2 - (
2 Sqrt[(a - k p)^2 (a^2 - 2 a k p + k^2 p^2 -
3 p \[Lambda]^2)])/\[Lambda]^2)^(3/4))/Sqrt[3] - (
I \[Lambda]^2 (3 p - (2 a^2)/\[Lambda]^2 + (
4 a k p)/\[Lambda]^2 - (2 k^2 p^2)/\[Lambda]^2 - (
2 Sqrt[(a - k p)^2 (a^2 - 2 a k p + k^2 p^2 -
3 p \[Lambda]^2)])/\[Lambda]^2)^(7/4))/(3 Sqrt[3])),
y -> -((I (3 p - (2 a^2)/\[Lambda]^2 + (4 a k p)/\[Lambda]^2 - (
2 k^2 p^2)/\[Lambda]^2 - (
2 Sqrt[(a - k p)^2 (a^2 - 2 a k p + k^2 p^2 -
3 p \[Lambda]^2)])/\[Lambda]^2)^(1/4))/Sqrt[3])}
..