Thank you Daniel and J.M.

Both pieces of advice were helpful. In my original solution for a certain M I had condA ~ 10^13 and condATA ~ 10^13. Artificial increase of precision (from MachinePrecision to 20) led to grows of condition number condATA by a factor of 10^3. Setting Tolerance option in SingularValueList equal to 0 made condATA increased condATA to 10^17. Value of condA remained (almost) unchanged.

By the way, applying both recommendations - increasing precision to 20 and setting Tolerance to 0 - results in brilliant correspondance of condATA and condA^2 (their ratio would be equal to 0.99999999999991). It looks the problem is solved.

Yes, as best I recall any error estimation is done differently e.g. using a condition number estimate.

I should add that it has been a very long time since I looked at the bignum linear algebra code.

Without the data you used to compute your matrix coefficients, it is hard to say anything substantial. Presumably you know the rule of thumb that you can lose up to $\approx \log_b(\kappa)$ base-$b$ digits in your solution?

In the meantime, here is a sketch of an experiment for you to try: use SetPrecision[] to (artificially!) increase the precision of your starting numerical data, and attempt the condition number calculations again; the results should ideally not be too different, and if they are markedly different, then you certainly have something worth investigating deeply.

Finally, you're computing and then throwing away the medium-sized singular values for computing $\kappa_2$; try using Norm[mat]/First[SingularValueList[mat, -1, Tolerance -> 0]] instead.