Message Boards Message Boards

GROUPS:

Changes in MathLink in Version 10 ?

Posted 5 years ago
16795 Views
|
26 Replies
|
16 Total Likes
|

In Version 10, MathLink is now called the "Wolfram Symbolic Transfer Protocol".

Is there new functionality or is it just a name change?

26 Replies

The interface level has been bumped to 4. If you look in mathlink.h, you'll find a list of new functions and a list of removed functions (quoted at the end). The functions can be looked up in the documentation. There's probably more to say here, and I'm curious about the changes too.

I also noticed that parallel kernels launch much faster in version 10, which is likely due to MathLink changes.


The name chage, in my opinion, is pretty counterproductive. Renaming all API functions (used for engineering) just because of a marketing decision (!!) to rebrand the system is just wrong and I don't see how it could bring aything but trouble ...

Now Mathematica comes with two separate directories, one for MathLink and one for WSTP. The two look identical other than the renaming. One contains the libraries and header files with the ML prefixed functions, the other one has the WS prefixed functions.


    /*
     * Interface 4 adds the following exported functions:
     *
     * MLCreateLinkWithExternalProtocol
     * MLDoNotHandleSignalParameter
     * MLEnableLinkLock
     * MLFilterArgv
     * MLGetAvailableLinkProtocolNames
     * MLGetInteger8
     * MLGetInteger8Array
     * MLGetInteger8ArrayData
     * MLGetInteger8List
     * MLGetLinksFromEnvironment
     * MLGetNumberAsByteString
     * MLGetNumberAsString
     * MLGetNumberAsUCS2String
     * MLGetNumberAsUTF16String
     * MLGetNumberAsUTF32String
     * MLGetNumberAsUTF8String
     * MLHandleSignal
     * MLIsLinkLoopback
     * MLLinkEnvironment
     * MLLogFileNameForLink
     * MLLogStreamToFile
     * MLLowLevelDeviceName
     * MLPutInteger8
     * MLPutInteger8Array
     * MLPutInteger8ArrayData
     * MLPutInteger8List
     * MLPutRealNumberAsUCS2String
     * MLPutRealNumberAsUTF16String
     * MLPutRealNumberAsUTF32String
     * MLPutRealNumberAsUTF8String
     * MLReleaseInteger8Array
     * MLReleaseInteger8List
     * MLReleaseLinkProtocolNames
     * MLReleaseLinksFromEnvironment
     * MLReleaseLogFileNameForLink
     * MLReleaseLowLevelDeviceName
     * MLReleaseParameters
     * MLSetThreadSafeLinksParameter
     * MLStopHandlingSignal
     * MLStopLoggingStream
     * MLValid
     * MLWaitForLinkActivity
     * MLWaitForLinkActivityWithCallback
     *
     * Interface 4 removes the following API functions.
     *
     ....
     */

What does "the interface level has been bumped to 4" mean?

I should have probably called it interface version 4, I'm not sure which is the official term used by WRI. It just means that the version number has been changed, which means that compatiblity with other interface versions is not guaranteed (new functions may be added, old functions removed or the calling syntax may change).

There is a seprate versioning for the public documented MathLink C interface and for the MathLink library. When the MathLink library gets a bugfix, the API to access it does not necessarily change. The API didn't change between Mathematica 8 and Mathematica 9, to my knowledge. But don't quote me on this, I'm not an expert on the topic.

You can find a helpful description of the changes in Interface 4 at http://reference.wolfram.com/language/tutorial/WSTPInterface4.html.

This is very useful. I didn't find this page myself.

Is there an option for mprep that causes it to output Interface 3 compatible code?

Also, have you seen this type of linking error with v10 on Linux? libML64i3.so: undefined reference to `uuid_parse'

Mprep should produce code that will work with the interface 3 version of the libraries. When building the code, if you want to link with the interface 3 library, you will need to define the preprocessor macro MLINTERFACE to 3, rather than letting it default to 4 in mathlink.h.

The linker error you are seeing that references uuid_parse is coming from a new requirement int he MathLink libraries to link with libuuid on Linux. You can use your Linux distribution's libuuid if you have one available or can install one from the distribution's package manager, or there is a libuuid in ${InstallationDirectory}/SystemFiles/Libraries/Linux-x86-64 suitable for use with the MathLink library.

Thank you for the response!

Will Installable MathLink programs compiled with the default MLINTERFACE 4 and linked against ML64i4 still work with earlier Mathematica versions?

For example, let's assume that we have the source code for an installable MathLink program that worked with some older version of Mathematica. Now I re-run mprep and re-compile the program, linking it against ML64i4 or ML32i4. Will the resulting binary be usable with older versions of Mathematica, e.g. Mathematcia 8 or 9?


I know that I can #define MLINTERFACE 3 and link against ML64i3 for compatibility, but the question is whether this is necessary?

The programs compiled and linked with the interface 4 version of the library should indeed work with older versions of Mathematica. I assume at some point you will run into issues such as older operating systems that will eventually result in incompatibilities the further you go back with your Mathematica revisions, but if you can run the two copies of Mathematica on the same machine, then your newer binary should work with the older Kernel.

Thanks for the reply! Then from now on I'll distribute binaries linked against interface 4.

Some more questions, just out of curiosity:

  • How does the new "IntraProcess" protocol perform for transferring large data? I'm asking because LibraryLink currently doesn't allow returning complex datatypes. For example, it's not possible to return both a matrix and a tensor from the same function, and returning them via multiple LibraryLink functions has the potential to introduce a mess (it forces me to maintain a library state, which isn't a good fit for Mathematica as it prefers side-effect-less functions). I think transferring very large packed arrays was already quite fast with MathLink with the "SharedMemory" protocol, but I'm not sure about many small arrays (or the overhead for each transfer).

  • Did MathLink receive any performance improvements in version 10, and generally does it have any advantages to link against interface 4 if I'm not using any interface 4 only features?

  • The IntraProcess protocol is designed specifically for communication between threads in the same process. The protocol should be quite fast.

  • MathLink did receive performance enhancements in Version 10 with respect to string and symbol transfers. These enhancements are automatically enabled with communicating with another Version 10 MathLink library and disabled when communicating with an older version of the library.

MathLink did receive performance enhancements in Version 10 with respect to string and symbol transfers. These enhancements are automatically enabled with communicating with another Version 10 MathLink library and disabled when communicating with an older version of the library.

Are these performance enhancements present even if I compile with -DMLINTERFACE=3 -lML64i3 (but using the libraries shipped with v10)?

In other words, does linking against libML64i4 have any benefits if I do not actually use any of the interface version 4 functions?

Could you please take a look at this question?

Some LibraryLink code I wrote using MathLink to transfer data still works in Version 10, even though the functions have been deprecated. I imagine that means it won't work when Version 11 comes out.

The LibraryLink interface that includes MathLink is very much tied to the interface used internally by the Kernel. If the Kernel is using MathLink interface 4, then the LibraryLink dlls need updated to support interface 4. The deprecated functions may continue to work for a time, however, as all things deprecated go, WRI may eventually remove those API functions from the interface altogether in a future release.

@SteveWilson

I'm right now investigating a bug report for MATLink on Linux. It appears that cycling a string through MATLink appends a zero character code at the end somehow, on Linux only (but not on OS X). I still haven't gotten to the bottom of the issue, and it is possible that I'm using the MathLink API incorrectly, but it looks more like a bug. I'm going to report it once I have had the time to create a minimal test case, but I wanted to give you advance warning.

enter image description here

The zero is likely appended when sending the string through MathLink the first time, not when receiving it back the Mma kernel.

The sending function is as simple as this:

void eng_make_String(const unsigned short *str, int len, int characters) {
    mwSize mbDims[2] = {1, len}; // use len, not characters, because no support for 4-byte characters in either Mma 9 or MATLAB
    mxArray *var = mxCreateCharArray(2, mbDims);
    std::copy(str, str+len, (unsigned short *) mxGetChars(var));
    returnHandle(var);
}

with template:

:Begin:
:Function:   eng_make_String
:Pattern:     engMakeString[string_]
:Arguments:     {string}
:ArgumentTypes: {UTF16String}
:ReturnType:    Manual
:End:

(You can actually ignore the guts of the functions, as it's MATLAB-specific code, but I thought a bit of context is always good. Also, I'm not sure it's still true that MathLink doesn't support UTF16 characters that need to be stored on 4 (not 2) bytes).

@SteveWilson

I haven't tracked the problem down fully yet, and I don't yet know why it only affects MATLink on Linux but not OS X.

But it looks like it's related to the fact that now transferring an UTF16String requires a byte order mark at the beginning. I should have read through the documentation page you linked to to notice this change, though maybe it would be good to have a documentation page dedicated to incompatible changes, or "how to upgrade".

Because there's the byte order mark, now the "length" of a UTF8String is one greater than before. The problem appears when trying to use MLINTERFACE=3, as the "length" is still one greater, but now there is no byte order mark!

Here's an example, tested on OS X with v9 and v10, compiled with the v10 MathLink libraries.

:Begin:
:Function:  cycle_string
:Pattern:   cycleString[s_String]
:Arguments: { s }
:ArgumentTypes: { UTF16String }
:ReturnType:    Manual
:End:

#include "mathlink.h"

void cycle_string(unsigned short *str, int len, int characters) {
    MLPutFunction(stdlink, "List", 4);
       MLPutUTF16String(stdlink, str, len);
       MLPutInteger16List(stdlink, str, len); // sloppily ignore signed/unsigned difference
       MLPutInteger32(stdlink, len);
       MLPutInteger32(stdlink, characters);
}


int main(int argc, char* argv[]) {
    return MLMain(argc, argv);
}

When I compile and link with interface version 4, this is what we get, both in v10 and v9:

In[3]:= cycleString["asd"]
Out[3]= {"asd", {-257, 97, 115, 100}, 4, 3}

When I compile and link with interface version 3, i.e. -DMLINTERFACE=3 -lMLi3, but use the libraries that come with v10, then I get this:

In[3]:= cycleString["asd"]
Out[3]= {"asd\.00", {97, 115, 100, 0}, 4, 3}

The "\.00" part is not displayed in the front end, it's invisible. It can only be detected with ToCharacterCode. This result is incorrect and I believe that it is a bug. It definitely causes incompatibilities with MathLink programs written for v9.

When I compile with the libraries that come with v9, I get:

In[3]:= cycleString["asd"]
Out[3]= {"asd", {97, 115, 100}, 3, 3}

For some reason this broke MATLink on Linux only but not on OS X, I'm not yet sure why. I am not yet sure if it is MATLink/MATLAB that's different between Linux and OS X or there's also a difference in the MathLink libraries that come with v10.

I have not yet reported this to Wolfram Support. I would like to wait for your input first.


Update: I have to correct myself: on OS X it might affect MATLink too, but we distribute compiled binaries on OS X, and it seems I only tested with v9-compiled binaries in v10 final. On Linux MATLink is always compiled from source so the problem came out.

Szabolcs,

I am very interested in working with this issue, however a death in the family will prevent me from intensively scrutinizing the issue until next week when I return from dealing with family issues. If you can isolate the problem prior to that time, please feel free to submit the issue to Wolfram Support, otherwise I will dive into the problem as soon as I return. I don't normally share private details on public forums, but I wanted you to know why I will not be answering this question for a period of time.

Thanks,

Steve

Steve, I am sorry for your loss.

I will summarize my findings and send them to support next weekend, so you can receive them next week.

Szabolcs,

I just wanted to confirm for you that you have indeed found a bug. I am working on applying a fix now.

Thanks,

Steve

Hi Steve,

Could you please clarify the precise meaning of length and characters for both interface version 3 and version 4 when passing an UTF16String? (as defined here, under Details)

Specifically, under what circumstances will these two lengths differ from each other? (Assuming that the bug is fixed, I'm not asking about the wrong behaviour.)

UTF16 can store a single character either on 2 bytes or 4 bytes. Is this the only reason why characters can differ from length?

Does MathLink support 4 byte characters with interface version 3? What about interface version 4? And what about Mathematica itself? (I could imagine that MathLink already supports 4 byte characters, but Mathematica doesn't yet.)

I am not very familiar with UTF16 and surrogate pairs, and I want to avoid dealing with surrogate pairs as much as possible in MATLink because (I think) MATLAB doesn't support them anyway. The reason why I am asking these questions is so that I can avoid any crashes or misbehaviour in my program when length and characters don't match.

In both Interface 3 and Interface 4 the length parameter indicates the number of bytes used to contain the character string. The characters parameter indicates the number of Unicode characters encoded in the string. These two values will differ precisely when dealing with UTF-16 surrogate pair characters. The characters parameter is given as a convenience value for the user of the API so that they do not have to scan the string returned from MathLink to verify the number of characters in the string.

MathLink fully supports the Unicode standard when transmitting UTF-8, UTF-16, and UTF-32 encoded character strings. It supports these transmissions between both Interface 3 and Interface 4, although you will need to take note in the change in semantics of the functions by the inclusion of byte order marks in the Interface 4 strings. Also to be completely clear, both Interface 3 and Interface 4 have full support for the UTF-8, UTF-16, and UTF-32 encoding forms.

As of this writing, Mathematica itself does not yet internally support the full range of Unicode characters, but instead supports a subset known as UCS-2.

If you want to avoid dealing with UTF-16 altogether, you can skip the Unicode functions and use the UCS2 functions. MathLink will handle all encoding conversion issues for you and for now you don't have to worry about the Kernel sending anything in the surrogate pair range of characters. That of course will almost certainly change in the future.

Thank you for the clarification. It sounds like I should have been using USC2String form the beginning.

If Mathematica will get support for the full Unicode range in the future, that will be a very nice addition. I have needed characters from outside the basic multilingual plane in the past, while dealing with a Chinese character decomposition table. At that time I had to parse the surrogate pair manually.

Posted 4 years ago

Is this change from MathLink to "Wolfram Symbolic Transfer Protocol", the reason why I'm getting this error msg from Mathematica Link:

"MathLink libraries are not installed or outdated. Before linking, you must install Mathematica or the latest version of the MathLink libraries."

I have installed Mathematica 10..2.0.0, Excel 2013 32-bit and MLinkforExcel_3.6.0. Inside Excel. ExcelLink works fine inside Mathematica but MathLink is not accessible from inside Excel.

Any help would be appreciated. Thanks.MathLink error msg

I might be completely wrong about this, but I seem to recall reading somewhere that the MathLink dynamic libraries (DLLs) used to be installed system-wide (in C:\Windows\system32 maybe?) before, but they are not since some recent version of Mathematica. You might try finding the libraries (presumably i3 versions, not i4; also make sure you copy the MathLink ones, not the WSTP one) in the Mathematica installation directory, and copying them next to the ExcelLink executable, if it has one.

I have never worked with ExcelLink, so I am not familiar with how it works.

To answer your question: I don't think this is because of the renaming to WSTP. The ML-prefixe version still ships with Mathematica and new versions of Mathematica are still compatible with old MathLink programs.

Posted 3 years ago

Thanks, that worked! I added this directory to the PATH variable:

enter image description here

Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract