a. Your two constraints are fine. NMinimize will search within and up to the edge of the region that you have specified and return the best solution it can.
b. I do not believe it is possible to add a constraint to NMinimize for the maximum error. It is possible to use NMinimize for a different function.
In[1]:= sol = NMinimize[{Max[
(a b Sech[266 b] - 0.391)^2, (a b Sech[128.3 b] - 0.3169)^2, (a b Sech[158.5 b] - 0.3969)^2,
(a b Sech[240.25 b] - 0.3117)^2, (a b Sech[156.8 b] - 0.6311)^2, (a b Sech[136.6 b] - 0.2162)^2,
(a b Sech[363.95 b] - 0.2727)^2, (a b Sech[106.35 b] - 0.3987)^2, (a b Sech[54.3 b] - 0.3596)^2,
(a b Sech[127.9 b] - 0.5211)^2, (a b Sech[179.7 b] - 0.27)^2, (a b Sech[105.3 b] - 0.889)^2],{a>100,b>.006}},{a,b}]
Out[1]= {0.100338, {a -> 107.292, b -> 0.00672948}}
(* Look at the size of the errors *)
In[2]:= {a b Sech[266 b] - 0.391, a b Sech[128.3 b] - 0.3169, a b Sech[158.5 b] - 0.3969,
a b Sech[240.25 b] - 0.3117, a b Sech[156.8 b] - 0.6311, a b Sech[136.6 b] - 0.2162,
a b Sech[363.95 b] - 0.2727, a b Sech[106.35 b] - 0.3987, a b Sech[54.3 b] - 0.3596,
a b Sech[127.9 b] - 0.5211, a b Sech[179.7 b] - 0.27, a b Sech[105.3 b] - 0.889} /. sol[[2]]
Out[2]= {
-0.15645, 0.200137, 0.0474598, -0.0358702, -0.182727, 0.28068,
-0.148911, 0.171069, 0.316762, -0.00309119, 0.125682, -0.316762}
Notice that the largest error in this is smaller than the largest error in my previous post,
but the other smaller errors are now somewhat larger with this method.
You should decide exactly what is the most important item that you wish to minimize.
There may be no solution with error<.001.
c. Any expression can be used in place of square if it gives a measure of how good the result is for your variables and the expression considers a value smaller than the desired to be as undesired as a value greater than the desired. You could use absolute value Abs[a b Sech[266 b] - 0.391] or fourth power or any other function that is greater than or equal to zero and probably symmetric around zero.