Did you clear all the variables?

data = Table[{x, 1/x}, {x, Range[6]}];

val = (a x^2 + b*x*y + c y^2 + d*x + e*y + f)^2 /. 
   Table[Thread[{x, y} -> data[[i]]], {i, Length[data]}];

In[16]:= NSolve[
 Thread[D[Plus @@ val + \[Lambda] (b^2 - 4*a*c + 1), {{a, b, c, d, e, 
      f, \[Lambda]}}] == 0], {a, b, c, d, e, f, \[Lambda]}, Reals]

Out[16]= {{a -> 0.071953, b -> 0, c -> 3.47449, d -> -0.922589, 
  e -> -6.58444, 
  f -> 3.96278, \[Lambda] -> 0.00264096}, {a -> -0.071953, b -> 0, 
  c -> -3.47449, d -> 0.922589, e -> 6.58444, 
  f -> -3.96278, \[Lambda] -> 0.00264096}}

You want the solution where f is negative.

I think Grad is a new function in Version 10. Try

In[2]:= NSolve[
 Thread[D[x + y + \[Lambda] (x^2 + y^2 - 1), {{x, y, \[Lambda]}}] == 
   0], {x, y, \[Lambda]}]

Out[2]= {{x -> 0.707107, 
  y -> 0.707107, \[Lambda] -> -0.707107}, {x -> -0.707107, 
  y -> -0.707107, \[Lambda] -> 0.707107}}

Here's an example of finding the minimum and maximum points for an objective function = x + y and a single equality constraint x^2 + y^2=1

In[42]:= NSolve[
 Thread[Grad[x + y + \[Lambda] (x^2 + y^2 - 1), {x, y, \[Lambda]}] == 
   0], {x, y, \[Lambda]}]

Out[42]= {{x -> 0.707107, 
  y -> 0.707107, \[Lambda] -> -0.707107}, {x -> -0.707107, 
  y -> -0.707107, \[Lambda] -> 0.707107}}

Well, this really isn't a site for having people write code for your projects or homework. So I'll outline it. Your problem is to minimize a sum of squares involving the quadratic parameter variables {a,b,c,d,e,f} where the quadratic is given by a*x^2+b*x*y+c*y^2+d*x+e*y+f==0. Your constraint is b^2-4*a*c+1==0. Define the function g[a,b,c]=b^2-4*a*c+1. So your objective function (the sum of squares), call it obj[a,b,c] (we ignore that it is a function also of variables {d,e,f}), must satisfy grad(obj) == lambda*grad(g) (where I use grad(...) to denote "take the gradient of ... with respect to the stated set of variables").

You can get seven equations in the seven variables {a,b,c,d,e,f,lambda} from:

g==0 (one eqn) grad obj == lambda*grad g (three eqns, since this we equate vectors of three components) D[obj,d]==0 D[obj,e]==0 D[obj,f]==0

These will be polynomial equations so one can use NSolve to solve the system.

This is about as complete an answer as I am inclined to give. More than that and I'd be doing a substantial part of your project.

My advice is to use a Lagrange multiplier and call it a KKT multiplier. Nobody will be any the wiser. The Lagrange multiplier, used as I noted, will give ellipses.

Yes, I've read. Whether these two methods can combine with KKT? Because i need that.

want to make my application was intact, and add in the menu KKT. That's the part I have to take in the menu?

 nearCoords = nearpoints[[All, 1]];
     {xs, ys} = Transpose[nearCoords];
     newellipse = aa*x^2 + 2*bb*x*y + cc*y^2 + 2*dd*x + 2*ff*y + gg;
     distance = Plus @@ (newellipse^2 /. {x -> xs, y -> ys});
     {res, coes} = NMinimize[distance, {aa, bb, cc, dd, ff, gg}];
     scaleup = FromDigits[{{1}, Last@RealDigits[1/(gg /. coes)] + 1}];
     esolve = Expand[scaleup*(newellipse /. coes)];

I suspect your professor is not going to be satisfied with your using the code I wrote. For one thing, I wrote the code to be as compact as possible, not as understandable as possible. I also wrote it so that the subscripts on the multipliers are the constraints they refer to. This is something that is easily done in Mathematica but very nonstandard. Since there is only one constraint, constructing the KKT equations is straightforward. I suggest you do that yourself for your code.

I modified my code to use NSolve instead of Reduce for this application.

data = Table[{x, 1/x}, {x, Range[6]}];
val = (a x^2 + b*x*y + c y^2 + d*x + e*y + f)^2 /. 
   Table[Thread[{x, y} -> data[[i]]], {i, Length[data]}];

consconvrule = {x_ >= y_ -> y - x, x_ == y_ -> x - y, 
   x_ <= y_ -> x - y, lb_ <= x_ <= ub_ -> (x - lb) (x - ub), 
   ub_ >= x_ >= lb_ -> (x - lb) (x - ub)};

KKTN[obj_, cons_List, vars_List] := 
 Module[{convcons, gradobj, gradcons, \[CapitalLambda]}, 
  convcons = (cons /. consconvrule);
  {gradobj, gradcons} = D[{obj, convcons}, {vars}];
  \[CapitalLambda] = Subscript[\[Lambda], #] & /@ cons;
  NSolve[Flatten[{Thread[gradobj == \[CapitalLambda].gradcons], 
     Thread[\[CapitalLambda]*convcons == 0] /. 
      Subscript[\[Lambda], Equal[a_, b_]] -> 0, 
     cons, {objval == obj}}], Join[{objval}, vars, \[CapitalLambda]], 
   Reals]]

resn = KKTN[Plus @@ val, {b^2 - 4*a*c <= -10^-3}, {a, b, c, d, e, f}]
{{a -> -0.00227535, b -> 0, c -> -0.109873, d -> 0.0291748, 
  e -> 0.208218, f -> -0.125314, 
  Subscript[\[Lambda], b^2 - 4 a c <= -(1/1000)] -> -0.00264096, 
  objval -> 2.64096*10^-6}}

Show[ContourPlot[
  Evaluate[a x^2 + b*x*y + c y^2 + d*x + e*y + f /. resn[[1]]] == 
   0, {x, 0, 16}, {y, -4, 4}], ListPlot[data]] gives an ellipse which passes through the points.

Why do you want to use the KKT conditions when it's simpler to use NMinimize with the constraint b^2- 4ac <= - epsilon, where epsilon is a small number, like 10^-3 ?

That's code I wrote to construct the Kuhn-Karush-Tucker conditions and solve them with Reduce. It's somewhat nonstandard as it finds maxima and minima simultaneously. It's designed for small problems which can be handled symbolically.