I do realize that this might be controversial, but I believe I have a real need for it this time.
If a process tries to access memory at a location it doesn't own, it will by default exit with a "segmentation fault" error. LibraryLink functions run in the kernel process, so if they crash, the whole kernel crashes. Mathematica is normally used interactively, so a kernel crash can be very unpleasant and lead to lost data ... (even if the front end doesn't crash and the notebook isn't lost).
How can I install some sort of protection so that when my LibraryLink function crashes, the kernel would print a warning about potential inconsistent state, but instead of exiting right away, attempt to give the user an opportunity to save (Export) their work?
Why do I believe that this is possible? Because MATLAB does it and sometimes R does it. Here's the dialog that MATLAB shows when a MEX function (equivalent of LibraryLink) crashes. Notice the Attempt to Continue button.
Why do I want it? I understand that the right solution is to just fix the crash. Or if that's really out of my control, then use MathLink and make it run in a separate process. But consider a situation like that of IGraph/M. It's a big LibraryLink project that can't be easily transferred to MathLink. It's using a large and complex library I didn't write myself that can crash in unexpected situations. That small window of opportunity for the user to save their work (when the kernel state is not completely corrupted) would really be a big improvement. igraph is not particularly crashy, but it does crash more often than builtin Mathematica functions.
What about catching SIGSEGV? My understanding is that on Unix systems it's possible to install a handler for SIGSEGV. Is this the right way to go? I don't want to catch all crashes, only those ones that are localized to my library. Is this possible? Doing a global modification to the kernel process seems like a bad idea.
Any solutions are going to be system-specific. I'm primarily interested in OS X solutions (but also Linux and Windows).
Cross posted on Mathematica.SE
