Thanks for the comments! It is encouraging that work is being done towards this goal.

Hi @Itai Seggev and @Dorian Birraux,

There seem to be basically three different efficient representations of byte sequences: strings, byte arrays, and byte-type rank-1 RawArrays.

Only RawArrays can be exchanged with C code nicely. (Strings are supported in LibraryLink, but handling them is cumbersome, and they are assumed to be null-terminated.)

I found that a ByteArray can be converted to a RawArray:

In[4]:= ba = ByteArray[Range[10]];

In[5]:= RawArray["Byte", ba]
Out[5]= RawArray["UnsignedInteger8", "<" 10 ">"]

What about the reverse? How can I convert a rank-1 byte-type RawArray into a ByteArray without unpacking it first into a list of 64-bit machine integers (and blow up the storage requirements 8-fold)?

Thanks for the response Dorian! In the meantime I also got a response on StackExchange, which pointed out that the type specification "ByteArray" can be used in LibraryFunctionLoad. In C code, it can be treated as a byte-type rank-1 RawArray. In Mathematica it will be a ByteArray. Thus one can write a simple library function that just returns a RawArray that was passed to it, but load it as LibraryFunctionLoad[..., {"RawArray"}, "ByteArray"].

I am looking forward to all this functionality becoming documented and brought to completion!

Do you mean the reverse, i.e. that not all Strings are a valid byte sequence? This gives True:

Let me rephrase what I wrote, I should have been clearer. You can indeed represent any unicode character codepoint from 0 to 65535 in a String. Internally though, characters are encoded on bytes, which requires to define a character encoding (i.e. a consistent way of representing values from 0 to 65535 using bytes). Given a character encoding, some byte sequences may be invalid.

e.g: In UTF-8 192 is not a valid byte and FromCharacterCode[192, "UTF-8"] returns an error.

From this follows that when you build a String out of bytes, its content is encoded. That's not required with byte arrays. I hope it clarifies.

So this can be seen as a kind-of PackedArray, but just for bytes? Semi-related, what about HDF5 import/export? It would be great to directly import towards a ByteArray.

How do I ask my int8 matrix to be converted to a ByteArray? I can't find an example...