As far as I know, the stand way for attacking this type of problem is unfolding the cone to a sector:

In the new coordinate system the coordinate of A and B is

p@A = {60, 0}; p@B = FromPolarCoordinates@{50, (2 π 20)/60};

and apparently the parametric equation for the shortest tour is

eq2D = (p@A - p@B) t + p@B // ToPolarCoordinates

where 0 < t < 1.

Then we just need some coordinate transform to find the equation in original coordinate system:

α = ArcSin[20/60];    
h = 60 Cos@α;
eq3D = Function[{r, θ}, 
   CoordinateTransform[ 
    "Cylindrical" -> "Cartesian", {r Sin@α, 60 θ/20, h - r Cos@α}]] @@ 
  eq2D

Graphics3D@{Opacity[0.5], Cone[{{0, 0, 0}, {0, 0, h}}, 20]}~Show~
 ParametricPlot3D[eq3D, {t, 0, 1}, PlotRange -> All]

Not sure if this problem can be attacked in a more straightforward way i.e. directly finding the equation in 3D space.

Hello Neil, thanks so much for the suggested solution, I am learning from your idea behind it, appreciated.

When i formulated the code, i.e. my very first idea of implementing the youtube problem statement in any kind of WL code form, I was not sure that the software could solve it. And indeed, it seems that Solve and NSolve cannot solve this problem when it is formulated like this, imho idiomatically. Maybe with a different coding approach, the two functions can solve it, who knows. But then i remembered Frank's method of solving an intricate set of equations .. simple use an optimization function with a pseudo objective function, taking the intricate set of equations as constraints. I find it more important nowadays to know how to solve a given non-trivial quiz/problem with a preferred software (e.g. Excel, Matlab, Raspberry Pi, Wolfram L, Maple, Python) than to solve it using the school-taught traditional brains-pencil-paper method. I'd rather invest time, brains, and efforts in finding a general coded solution than trying to solve the problem manually on a piece of paper (which i would later toss in a trash can). Both methods can be fun but imho there are more practical skills (like debugging) and experience to be acquired from setting up a working code; and it is easier to store/save/keep the code neatly (e.g. for future reference) because it is not written on a piece of scrap paper. Well, our way of keeping/collecting such code could be by posting them all in this thread haha.

@xzczd , amazing, I feel beaten :D — Thanks for the great solution showing that Solve can solve the problem but only after splitting the areas into triangles!

I'm not sure when it's improved, but at least since v12.3.1, we no longer needs to split into triangle!:

area[pts_] := Area[Polygon[pts], Assumptions -> {0 < x < a, 0 < y < a}]

o = {0, 0}; A = {a, 0}; B = {a, a}; F = {0, a};
G = (o + F)/2; H = (o + A)/2; J = (A + B)/2; L = (F + B)/2;
K = {x, y};
a^2 - 16 - 20 - 32 /. 
 Solve@{area[{o, H, K, G}] == 16, 
        area[{G, K, L, F}] == 20, 
        area[{J, K, L, B}] == 32,
        a > 0}
(* {28} *)

Oops, I see you already figured that out -- the post has been updated from the email version.