This was a bit tricky. I had to read the Trace pretty closely to figure it out.
Times[x___]
turns into just x___
.
So your code becomes:
Replace[Times[a, b], x__ :> Dot[x]]
Now x___ matches the entire expression so x is matched to Times[a, b]. This evaluates to
Dot[Times[a, b]]
Which in turn becomes just:
Times[a, b]
The key to preventing things like this is to use HoldPattern:
Replace[Times[a, b], HoldPattern[Times[x___]] :> Dot[x]]
The lesson I think is that it's dangerous to mix pattern matching with expressions which might evaluate on the patterns.