I installed Mathematica (12.0.0) in my HPC environment, and after getting an X-Windows server going on my laptop, was able to run it and just did some rudimentary things. This was run on one of the login nodes - the wrong place for it in production, I know, but I was just testing. Eventually I decided to exit and work out making sure regular user access to the program environment would work. But some sort of error came up. The popup that had the message was garbled, so I couldn't read it. But I also couldn't close the window or its ancestors. I eventually exited from my Linux session (CentOS 6.x), and even that didn't seem to kill it. I had to go back to my desktop and exit from X to close these windows and (I thought) exit from Mathematica.
Since then, I went to log in again and got the following: -bash: fork: retry: Resource temporarily unavailable -bash: fork: retry: Resource temporarily unavailable -bash: fork: retry: Resource temporarily unavailable -bash: fork: retry: Resource temporarily unavailable -bash: fork: Resource temporarily unavailable I didn't realize it, but Mathematica was still running. When I finally figured that much out, I killed the individual Mathematica processes that were running, and was able to reconnect to the server and login normally.
The question really is, has anybody experienced this sort of thing? If so, how did you resolve it? I obviously can't turn this over to the user community with this sort of defect or I will spend the rest of my days killing mathematica.
I will eventually need to get into recommended ways to configure X and Mathematica. But for now, I just need to be able to keep this problem from coming up. Any help?