Because mats still has the value "1". It will get unstuck when you type a new value other than "1" in the InputField and hit enter. Now the loop will stop and the button will go away when you click OK on it.

How about the Null part of my question? Any way to suppress it?

For the second part, I do not see a Null printed, but the actual If statement itself

The reason is because that is how If works when it has one branch and it is not true. For example

To workaround this, use the third branch of If which tells it what to do if it can't decide between True and False

So for your first example, to eliminate the Null you can type in a value for mats other than "1", is to give it a value to use for the second branch. Also, it is best to initialize mats to "1" for first time, so the code becomes

mats = "1";
InputField[Dynamic[mats], String]
Dynamic[If[mats == "1",
  CreateWindow[DialogNotebook[{TextCell["Not a valid matrix."], DefaultButton[DialogReturn[]]}]]
  , "waiting..."], TrackedSymbols :> {mats}]

You do not need a value for the third branch (the undecided one) since now mats is initialized to some specific value.

Not really. Dialogs are not well explained in context of Dynamic or different links. For example, this one is not reevaluated:

x = 1;     
Dynamic[ If[x == 1, Print[DateList[]]; 1, 2], TrackedSymbols :> {x}]

I don't know why it behave differently.