i'm not a win10 user however "resource limits" can be set per application (and per user) on any modern OS. that means mathematica would not be able to hog (disk, cpu, maybe other) resources you need to keep your computer responsive.
as far as the mathematica notebook doing the calculation it would normal for it to appear not to respond
however, there are keys that "interrupt OR abort" the current calculation
the calculation may generate huge data quickly and be transferring the data to the front end (viewer window, which cannot take and view that much data quickly or at all). in the older mathematica (ie, version 4) the only remedy was to 'xkill' the front end and connect another (advanced usage) or quit and re-open mathematica
imho it should be easier than it was for mm 4.0