You're welcome. I asked about my "wonderment" earlier because I had noticed that the following ways also produce the desired form, either from 2D or from 3D vectors. But I wasn't sure that we could assume the equation would always have this form.
eqn = (x1^2 x2^2)/a^4 + (x2^2 y1^2)/(a^2 b^2) +
(x1^2 y2^2)/(a^2 b^2) + (y1^2 y2^2)/b^4 == 1;
(*1*)
scale = {a, b};
v = {x1, y1}/scale;
w = {x2, y2}/scale;
Simplify@Det[{v, w}]^2 + (v . w)^2 == 1
(*2*)
scale = {a, b, 1}; (* 3rd component does not matter in this case *)
v = {x1, y1, 0}/scale;
w = {x2, y2, 0}/scale;
Simplify[# . # &@Cross[v, w]] + (v . w)^2 == 1
This implies the solutions may be derived from scaled vectors v, w whose lengths multiply to
$1$,
since Det[{v, w}]^2 + (v . w)^2 equals Norm[v]^2 Norm[w]^2.
Based on the above, here are two similar ways to derive the desired form from the expanded polynomial:
scale = {a, b};
v = {x1, y1}/scale;
w = {x2, y2}/scale;
reduction = {A == Det[{v, w}], B == v . w};
1 == 1 + First@GroebnerBasis[Join[{eqn}, reduction],
{x1, y1, x2, y2, A, B}] /.
First@Solve[reduction, {A, B}]
Eliminate[Join[{eqn}, reduction], {x1, y1, x2, y2}] /.
First@Solve[reduction, {A, B}] //
AddSides[#, 1] &
The second way might be better. It seems to return True if eqn does not satisfy the hypothesis (that eqn is of the form to be expressible in terms of A and B). GroebnerBasis[] will return a polynomial that will not be in the right form, if eqn cannot be put in the right form; one should check it before substituting A and B.