Don't you have any other choices in the Select compiler combo box (see attached image)? SystemModeler should auto detect your visual studio installation and present it as a choice there.
There is a bug that in 4.0 that if you select Custom and then browse for the directory to find vsvars32.bat it will always say that it fails to detect compiler version. However, it should work to manually select the compiler version and then OK. You can verify that it worked by trying to simulate IntroductoryExamples.HelloWorld for example.
Also, note that the "Path to vsvars32.bat" should be the path to the directory where vsvars32.bat is located. So in your second example the correct path is C:\ProgramFiles (x86)\Microsoft Visual Studio 11.0\Common7\Tools (no \vsvars32.bat at the end).