We've upgraded Mathematica from 8.0.4 to 10.0 and now 10.0.1. Running on a 64bit Debian Wheezy-Server, sometimes about every 3rd startup sees MathKernel dying (there are 2 at startup, one MathKernel -sandbox, one MathKernel -mathlink; the later dies), and in the console we get
LinkConnect::linkc: -- Message text not found -- (LinkObject[btkfn_shm, 3, 1])
(The symbol btkfn_shm changes each time, of course.) The frontend continues to work, however, evaluations are not possible. If we start Mathematica and have it run an evaluation right away, apparently, short evaluations are done. But in general, until the kernel dies and we see the error message we can't do much more than enter 1 or 2 commands.
We tried it with ssh -X and within a VNC session. This seems to make no difference. A web research suggested to make sure the loopback interface is available, but that's the case. iptables exist but do not block localhost. We tried -clean, -cleanStart, and -dv, too. Mathematica 8.0.4 still works fine. Running only the kernel with math works also.
Could this be an issue with the license server (we connect to our university's license server) since it seems to happen more frequently at times!? Like during the day. And right now, 6 p.m., I had 30 startups with no problem. If it's related to the license server, how could I debug this?
math -lmverbose works fine. It doesn't crash and shows that it successfully gets and releases the licenses. Also, my colleagues who run the license server said that we are far from reaching the limits of available licenses.
The rate at which we observed the LinkConnect error and dying kernels suddenly dropped so much that we have to postpone further analysis. Worth noting might be that the only thing that we actually changed on the server right before problems went away almost completely was
rm -r /tmp/MathLink
Unfortunately, I didn't keep a copy of the rather messy directory. So we cannot test right now whether this was indeed a major cause. However, left-over files start compiling again. So, we'll see.
BTW, why the heck are these temporary files world-readable?
If it's related to the license server, start a Mathematica session, then launch a separate Kernel in a terminal window with
The -lmverbose will make it print the kernel's view of the traffic with the MathLM.
You might need to start the separate kernel multiple times to get something interesting. It would be helpful if the MathLM were logging its traffic, too. (MathLM command-line options are in http://reference.wolfram.com/language/ref/program/mathlm.html, under Details.)
If you do get a crash with the terminal Kernel, send us what information you have including error messages.