Group Abstract Group Abstract

Message Boards Message Boards

3
|
12.5K Views
|
15 Replies
|
11 Total Likes
View groups...
Share
Share this post:

LibraryLink back- and forwards compatibility broken in 10.4!

Szabolcs Horvát
Posted 10 years ago
POSTED BY: Szabolcs Horvát
15 Replies
Sort By:
Itai Seggev
Itai Seggev, WOLFRAM
Posted 10 years ago

I'm curious, why is the 4.25 (libstdc++) version included with 10.4? Is it for compatibility with certain MathLink programs which run as a separate process (but perhaps not for use with LibraryLink which runs in the kernel process)?

It is both for compatability with standalone MathLink programs and to allow you to build libstdc++-based LibraryLink libraries. A standalone executable linked against the framework will remember the "major version" (in this case, 4.25) it was linked against and find it here, allowing it to continue to run.

Given that it is included, is it still expected that the LibraryLink binary I compile with 10.3 does load in 10.4, but then crashes sporadically (so far I only see a crash when calling MLNextPacket())?

Yes. The advantage and drawback of LibraryLink is that it loads functions directly into the kernel process. The kernel is linked against the libc++ version of MathLink. The 10.3-built LibraryLink is linked against the libstdc++ version. When you load it, it will load its MathLink along with it. You now have two sets of the same function loaded in the kernel process, with differeing C++ runtimes, and chaos ensues.

Incidentally, the reason the 10.4 built library won't load in earlier versions is that it knows it needs at least version 4.36 of MathLink, which doesn't exist in a 10.3 layout and so the load operation fails. If there were some way to prevent a second copy of a framework from being loaded into the same process, we could prevent 10.3-built libraries from loading into 10.4 instead of crashing the kernel. But Apple's linker doesn't provide a mechanism to do that (at least as far as we know).
POSTED BY: Itai Seggev

Thank you, I'm going to try this! :)

I'm curious, why is the 4.25 (libstdc++) version included with 10.4? Is it for compatibility with certain MathLink programs which run as a separate process (but perhaps not for use with LibraryLink which runs in the kernel process)? Given that it is included, is it still expected that the LibraryLink binary I compile with 10.3 does load in 10.4, but then crashes sporadically (so far I only see a crash when calling MLNextPacket())?

This is more out of curiosity (and wanting to make sure that nothing is broken in the 4.25 version), as I already modified my package to have two binaries on OS X. I'm not looking to use a single one.
POSTED BY: Szabolcs Horvát
Itai Seggev
Itai Seggev, WOLFRAM
Posted 10 years ago

You can in fact build a 10.3-compatible version with 10.4, it's just slightly ugly. What you need to do is replace

-F"/Applications/Mathematica 10.4.app/Contents/SystemFiles/Links/MathLink/DeveloperKit/MacOSX-x86-64/CompilerAdditions" -framework "mathlink"

with

-Wl,"/Applications/Mathematica 10.4.app/Contents/SystemFiles/Links/MathLink/DeveloperKit/MacOSX-x86-64/CompilerAdditions/mathlink.framework/Versions/4.25/mathlink"

so that you're explictly linking in the libstdc++ version of MathLink rather than the libc++ version (called 4.36). This is not at all obvious from Apple's documentation, and we had to experiment internally to make sure this worked prior to shipping 10.4. I'm glad we did so that I had this answer handy for you. :)
POSTED BY: Itai Seggev

I believe that if you add the following extension to your PacletInfo.m, the $SystemID value is significant: 

Paclet[
        Name -> "My Paclet",
        Version -> "x.x.x",
        Extensions -> {
            {"LibraryLink", Root->"LibraryResources"}
        }
]

This will tell FindLibrary where to find your libraries (informed by the $SystemID). Using a full path is perfectly valid for corner cases like this, though.
POSTED BY: Alex Newman

Thank you for the help Alex! I'm now keeping two OS X binaries in LibraryResources/MacOSX-x86-64-libc++ and LibraryResources/MacOSX-x86-64-libstdc++ and all is working well. I don't think violating the convention about LibraryResources/$SystemID is a problem because I am adding it to $LibraryPath manually anyway (if the system didn't add it already).
POSTED BY: Szabolcs Horvát
POSTED BY: Alex Newman

The problem exists only on OS X

Yes, correct.

There's only one break in compatibility, i.e. all 10.x for x?3 and again all 10.x for x?4 are going to be compatible with each other too?

That should largely be true for the 10.x series of products, but I don't think we guarantee perfect binary compatibility between versions with MathLink or the Wolfram RTL.

(you seem to have replied elsewhere, so I'll reply there)

POSTED BY: Alex Newman

Hello Alex,

Thank you for your very quick answer!

You answered most of my questions from above. Only two remain:

Question 1:

If I only have M10.4 installed, can I compile a binary that will work with M10.3? To make the question more concrete, suppose I use the following command line (slightly modified from what CreateLibrary uses):

-mmacosx-version-min=10.6 -framework Foundation -I"/Applications/Mathematica 10.4.app/Contents/SystemFiles/IncludeFiles/C" -I"/Applications/Mathematica 10.4.app/Contents/SystemFiles/Links/MathLink/DeveloperKit/MacOSX-x86-64/CompilerAdditions" -F"/Applications/Mathematica 10.4.app/Contents/SystemFiles/Links/MathLink/DeveloperKit/MacOSX-x86-64/CompilerAdditions" -framework "mathlink" -lstdc++

I changed -lc++ to -lstdc++ and used -mmacosx-version-min=10.6 as before. But it's "using" (whatever that means here) the MathLink "framework" from 10.4, not 10.3. Will the result work with 10.3?

I don't plan on deleting 10.3 anytime soon, but I won't be keeping it forever ...

Question 2:

With Mathematica 10.3, I compiled some libraries (where I used C++11) using -mmacosx-version-min=10.9. Without this I couldn't use C++11 features ... The result did work so far. But could this, in principle, cause problem with 10.3 (which uses -mmacosx-version-min=10.6 by default)? Does -mmacosx-version-min=10.9 imply anything about libc++ vs libstdc++?
POSTED BY: Szabolcs Horvát

Thank you for the response. Does this mean then that:

  • The problem exists only on OS X
  • There's only one break in compatibility, i.e. all $10.x$ for $x \le 3$ and again all $10.x$ for $x \ge 4$ are going to be compatible with each other too?

Compiling only two binaries and only on one OS is not that big a deal.

A few more questions:

Is it possible to compile for <= 10.3 if I only have 10.4 available? I noticed that the only difference in the command lines is that 10.3 has -mmacosx-version-min=10.6 -lstdc++ while 10.4 has -mmacosx-version-min=10.9 -lc++. That part seems independent of Mathematica. But we also have -F"..../Mathematica.app/..../CompilerAdditions" and -framework "mathlink". I don't understand what a "framework" is exactly and what sort of linking it involves (static or dynamic). Thus if I specify M10.4's MathLink framework (not 10.3's!) when compiling and I use -lstdc++, will the result work with 10.3? (I know I should try this out...)

Are there similar compatibility breaks on other systems (perhaps between other already released 10.x versions)?
POSTED BY: Szabolcs Horvát

@Szabolcs Horvat This is an issue that affects Mac OS X only and should not occur again for the foreseeable future. As @Ilian Gachevski points out, this has to do with Mathematica switching C++ standard libraries on Mac. This is a one-time change we had to make following Apple dropping support for the GNU libstdc++, which was affecting our ability to use modern C++ features and link to modern interfaces and features targeting Mac OS X 10.9 and above.

Now, it might not seem like an issue if you're writing your LibraryLink code in straight C, but anything that the CCompilerDriver builds is going to be linked against MathLink, which is a C++ library. When you're loading LibraryLink code, your library is part of Mathematica's process and needs to basically conform to the same toolchain that Mathematica used-- basic C++ types can appear corrupt when passed between these separate C++ libraries. We use the CCompilerDriver to help users follow our platform as closely as possible (which is why they're producing a different binary in these two versions).

For this particular issue, I would recommend building one instance of your library with 10.3.x and another instance with 10.4 and loading the 10.3 instance if $VersionNumber is less than 10.4 to choose which file to load with LibraryLoad. Once more, this should not affect Linux or Windows. It also will not affect code outside the process (code built with MathLink and loaded with Install or built separately and loaded with ProcessLink).

Please let me know if you have any additional questions about this problem or if there's any other guidance I can provide on how to work around this.

POSTED BY: Alex Newman

@Szabolcs I am afraid maintaining two separate versions of the library on OS X may be unavoidable, this is caused by the libc++ vs libstdc++ ABI incompatibility.
POSTED BY: Ilian Gachevski

OK, here's the test case for the reverse problem:

https://dl.dropboxusercontent.com/u/38623/Backward-compat-testcase.zip

If I compile a LibraryLink library with 10.3.1, then certain things don't work correctly in 10.4.0. Specifically, MLNextPacket() crashes when used as shown here.

Reported as CASE:3544766.

I think it is absolutely essential for Wolfram Research to have a prerelease programme if they want to support a healthy add-on ecosystem. Add-on authors must be able to test their packages before a new version of Mathematica is released.
POSTED BY: Szabolcs Horvát

I haven't run the full testsuite for MATLink yet, but the few things I tried worked correctly. MATLink does not use LibraryLink. It only uses MathLink (WSTP).
POSTED BY: Szabolcs Horvát

Hi Szabolcs,

How about your MATLink package, does that use LibraryLink under the hood? I'm using it now, and will go to 10.4 soon as well... or should I perhaps keep both versions installed?

--SH
POSTED BY: Sander Huisman

Reported one of the two problems (libraries compiled with 10.4 don't load in 10.3) as CASE:3544676.

The test case I sent: https://dl.dropboxusercontent.com/u/38623/Forward-compat.zip

Still working on a test case for the other problem (which is much more involved).
POSTED BY: Szabolcs Horvát
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Tag limit exceeded
Note: Only the first five people you tag will receive an email notification; the other tagged names will appear as links to their profiles.
Reply Preview
Attachments
Remove
or Discard