How do I solve this problem

solve in R

(x+y)^2 = (x+4)(y-4)

You did not say what you are solving for. If solving for x then

Solve[(x + y)^2 - (x + 4) (y - 4) == 0, x]
{{x -> 1/2 (-4 - y - Sqrt[3] Sqrt[-16 + 8 y - y^2])}, {x -> 1/2 (-4 - y + Sqrt[3] Sqrt[-16 + 8 y - y^2])}}

If solving for y, then

Solve[(x + y)^2 - (x + 4) (y - 4) == 0, y]
{{y -> 1/2 (4 - x - Sqrt[3] Sqrt[-16 - 8 x - x^2])}, {y -> 1/2 (4 - x + Sqrt[3] Sqrt[-16 - 8 x - x^2])}}

You can’t really solve for both x and y, since there is only one equation and two unknowns. There should be the same number of unknowns as the number of equations.
If you try to solve for both, Mathematica will complain:

Solve[(x + y)^2 - (x + 4) (y - 4) == 0, {x, y}]
Solve::svars: Equations may not give solutions for all "solve" variable

And if you really insist on a solution, you can try

SolveAlways[(x + y)^2 - (x + 4) (y - 4) == 0, {x, y}]
{}