Message Boards Message Boards

WSTP: Handling lists

Posted 4 years ago

I am writing a WSTP application in C, and would like to "receive" a list from Mathematica, however, it should be able to handle multiple datatypes (i.e. int, real, etc.); is there a way to do this with a single function or I need to write a function per datatype, i.e.

processIntegerList(int *a, long b); progressReaList(double *a, long b);

If this can be achieved with a single function, could you paste a snippet of code that would then identify the datatype of the list passed (we assume here that the argument is always a list).

Thank you for looking.

POSTED BY: jerome ibanes
8 Replies

Actually, with LibraryLink, it is not necessary to fix either the element type (Integer, Real, Complex) or the dimension (1D, 2D, etc.) of either the input arguments or the return value. Is this flexible enough for your purposes?

POSTED BY: Szabolcs Horvát
Posted 4 years ago

Can you explain a bit more why you want to do this please?

I'm writing a set of functions in C which perform various computations (for multiple reasons C is a better candidate here for this specific set of functions than the Mathematica language). I had initially started my work with LibraryLink, however, couldn't figure out how to return arbitrary lists (which datatype isn't defined at compile time); some functions may return Real32Lists or Integer32Lists or else, it greatly varies on the arguments being passed as input. Likewise, the input(s) should be arbitrary, meaning those functions may be called with either Real32Lists or Integer32Lists or else. This problem went away when using "Manual" on WSTP; so I've used WSTP since. I intend to add Arrays at some point (for input/output), but only limiting the scope to Lists for simplicity in this thread.

With this being said, I will show an example of a function function1 takes two arguments, one string, one list (of arbitrary type). function1 then performs some form of computation on the arguments, and either returns something (likely a List or an Array, or else or nothing). Even if it doesn't returns anything, it may still have some data allocated in memory that another function might use to "pursue computation" if I may say.

Let me know if that makes sense, thanks for looking.

POSTED BY: jerome ibanes

I linked to a number of pointers in my previous posts: do those help?

POSTED BY: Szabolcs Horvát

While conceptually a List is just another expression with head List and arbitrary elements, the implementation uses more efficient representations (packed array) when all elements are either machine reals or machine integers. This representation is maintained when sending the array through a MathLink connection. In principle, you can (1) detect if there is a packed array on the link (2) determine its type (3) use the correct Get function to retrieve it.

Unfortunately, how to do this is not documented. Lots of things about MathLink are not very well documented. There is a question about this on M.SE: https://mathematica.stackexchange.com/q/180706/12

I would not recommend trying to do this, unless you have a very good reason to do so (e.g. you do not know at all what expression you are receiving on the link). Instead, you can try to determine the type of the array in Mathematica code, and either pass in the type in some way (some encoding of your choice) to the C functions, so that function knows what to get, or just use two different functions (what you were trying to avoid).

If you do this anyway, LibraryLink will give you much better performance: https://mathematica.stackexchange.com/q/91887/12 With LibraryLink, having two functions will not be avoidable.

Note that by using LibraryLink you will not lost almost anything because optionally you can still use MathLink-based passing.

To determine the array type in Mathematica, use Developer`PackedArrayQ (see 2nd argument). To determine if a non-packed array is suitable for passing to your function (i.e. if types are homogeneous, if they fit in machine numbers, etc.) you can first try to pack it using Developer`ToPackedArray.


All that said, if you all wanted was to save a bit of memory, then just don't specialize the functions to integer/real. It will not save memory at all. If you care about performance a lot, just use LibraryLink.

Let me also recommend my LTemplate package with the caveat that you should get familiar with plain LibraryLink before starting to use it. LTemplate is most useful if you want to use managed library expressions, i.e. if you want to store some sort of data in C and manipulate it from Mathematica. I use it for my IGraph/M package, where I need to store graphs in C and manipulate them repeatedly.


I hope this information will help you make a good decision.

POSTED BY: Szabolcs Horvát

The situation is a bit more complicated than you may be aware. If you explain the details and answer the granular questions I asked, I could give an answer that will be more practically useful for you.


Someone might give you this answer, which is not technically incorrect, but I do not think it is useful, mostly because it will be terribly slow:

A List does not have a datatype. Only its elements do. We can have a list with heterogeneous datatypes. You need to get each element in turn and determine their datatypes. You can use MLGetType to check the type of the first argument, assume that as the element type of your list, then iterate through the rest of the elements, check that they all have the same type, and get them one by one.

POSTED BY: Szabolcs Horvát
Posted 4 years ago

What I mean is that I want a ArgumentType of "Manual". Based on that I can WSCheckFunction() that the input is a "List" (this is what I want); but how can I find its datatype, and access each element, so I can parse it efficiently?

Why is just using reals not sufficient? Integer values can be stored in doubles.

uses twice as much memory

If yes, why are you using MathLink and not LibraryLink (which has much better data passing performance)?

Haven't found how to ReturnType "Manual" with LibraryLink, that is, I don't know the output of the function until it's run; I am not sure if LibraryLink can have a ReturnType as Manual like WSTP?

POSTED BY: jerome ibanes

I see that you have edited your post and answered the questions.

uses twice as much memory

This is not true. On a 64-bit platform, Mathematica uses 64-bit integers. A double is also 64-bit. This saves no memory.

Using LibraryLink instead of MathLink will save memory though because you can access packed arrays without making a copy.

Haven't found how to ReturnType "Manual" with LibraryLink, that is, I don't know the output of the function until it's run; I am not sure if LibraryLink can have a ReturnType as Manual like WSTP?

What kind of data do you return?

It is true that LibraryLink is inconvenient in that normal functions can only return a single result of a single type. The question I asked about returning multiple results is here: https://mathematica.stackexchange.com/q/31545/12

I usually do one of two things:

  1. Encode the actual return value into a format that LibraryLink can return. This can be as simple as concatenating two arrays of different lengths and appending their lengths, or as complicated as encoding everything into a format like JSON or WXF

  2. Use one or more separate LibraryLink functions to retrieve the results. Note that LibraryLink functions can use MathLink-based passing, which corresponds to the "Manual" passing you mentioned. But using this comes with a performance penalty again.

POSTED BY: Szabolcs Horvát

Can you explain a bit more why you want to do this please? Of course, this is a reasonable question, but the situation is a bit more complicated than you may be aware. The best solution will depend on what you are trying to achieve.

Why is just using reals not sufficient? Integer values can be stored in doubles.

Is this about performance? If yes, why are you using MathLink and not LibraryLink (which has much better data passing performance)?

POSTED BY: Szabolcs Horvát
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract