Assume a=b=1 and y[0]=1.
eq = y[n] - 1 - 1*Sqrt[y[n]] - y[n + 1];
sol = First@RSolve[{eq == 0, y[0] == 1}, y[n], n];
Table[(y[n] /. sol), {n, 10}]
(*{0. + 2. I, -3. + 4. I, -8. + 6. I, -15. + 8. I, -24. + 10. I, -35. +
12. I, -48. + 14. I, -63. + 16. I, -80. + 18. I, -99. + 20. I}*)
Check numericaly:
RecurrenceTable[{eq == 0, y[0] == 1}, y, {n, 10}] // N
(*{1., -1., -2. - 1. I, -3.34356 + 0.455347 I, -4.46779 - 1.37741 I, -5.78989 + 0.760707 I, -6.94763 - 1.65068 I, -8.25859 +
1.00344 I, -9.43286 - 1.87562 I, -10.7367 + 1.21067 I, -11.9212 - 2.07121 I} *)
Yes possible, a BUG.