The kernel used to use UCS-2 internally. It now uses a variant of UTF-8. MathLink also gained functions for sending and transmitting / receiving UTF-8. These were steps 1 and 2 in the process for getting the full unicode character set. But there are several additional steps to actually get there. Important additional ones include creating equivalents of \:wxyz for non-BMP characters; getting the kernel, MathLink, and FE successfully talking to each other using these new methods; and finding all in the places where the assumption that characters lie in the range 0-65535 is hardcoded, either implictly or explicitly, and updating the code. We have made progress on some of these internally, but as you might imagine its an on-going process and we certainly can't promise the feature on any particular timeline.

Please do not forget about Unicode in file paths:

• How to workaround failures with Unicode filepaths?

You can't directly create a ByteArray from a RawArray. But, I see no reason not to support it, since we already have ByteArray from PackedArray.

Thank you for the response Dorian.

... not all byte sequences are a valid String, so, when one stores bytes in a string, the data needs to be validated

Do you mean the reverse, i.e. that not all Strings are a valid byte sequence? This gives True:

tup = Tuples[Range[0, 255], {2}];
tup2 = ToCharacterCode /@ FromCharacterCode /@ tup;
tup2 === tup

All possible byte values, including 0, seem to be storable in Strings.

But either way, it is clear enough that a dedicated ByteArray is better for storing byte data than a string. That doesn't need to be explained further.

Other than the ones I mentioned, are there any operations we can perform on ByteArrays (especially other things than element extraction)?

Here are a few more suggestions, in addition to the ones I already mentions:

Efficiently changing elements in-place through Part:

a = ByteArray[...];
a[[2]] = 5;

(This should also support Span, i.e. ;;)

Append, Prepend, AppendTo, PrependTo.

Something like Partition to break a big array into parts. My envisioned use case is processing a large ByteArray without unpacking the whole thing to an integer list. Instead, we could unpack small sections at a time, process them, then re-pack them. So perhaps other methods, such as Map, BlockMap, etc. are more appropriate. (E.g., Audio has AudioBlockMap). If ByteCount can be trusted, there is a storage overhead of 96 bytes, so perhaps complete pre-Partition-ing is not the best.

Direct creation functions: Analogues of ConstantArray (large constant byte array) and RandomInteger (for random bytes).

But the most important missing functionality is conversion:

• to/from strings (like FromCharacterCode, ToCharacterCode)
• to/from files (BinaryRealList, BinaryWrite)
• and very importantly: LibraryLink. Conversion to/from RawArray would suffice, as RawArrays work with LibraryLink since version 10.4. This would allow us to implement efficient functions for anything we need. I looked into the implementation of some of the built-in functions, and I see that currently sending to/from LibraryLink is done through an inefficient conversion to a 64-bit integer list (i.e. {Integer, 1} LibraryLink type).

ByteArray represents bytes, internally using one byte per value which make them space efficient. Because the data are binary, the performance gain of using them in place of string of bytes, is significant. Indeed, not all byte sequences are a valid String, so, when one stores bytes in a string, the data needs to be validated. That not required with ByteArray.

There is an effort to implement most if not all the features you've listed as top level function. In the mean time, here are some non documented functions that you may find interesting, as a complement to those you already mentioned:

• Developer`EncodeBase64: ByteArray to String convertion, takes a byte array, returns a base64 string.

In[1]:= Developer`EncodeBase64[ByteArray[Range[5]]]
Out[1]= "AQIDBAU="

• BinaryWrite accepts ByteArray, as an undocumented feature. I'm not aware of a reverse function that reads a file directly into a byte array.

Generally speaking, ByteArray is recommended everywhere one uses binary data. It leads me to talk about BinarySerialize. BinarySerialize serializes any Wolfram Language expression to a binary representation that is platform independent and fast to deserialize. Contrary to MX which contains all the definitions of a given expression, like DownValues, BinarySerialize is more data oriented, and only somehow represents the FullForm of an expression. The format used by BinarySerialize has simple enough specifications that we may consider publishing them.

As users noticed on StackOverflow, BinarySerialize does not always produce a smaller output with respect to ByteCount. One reason to that is, contrary to Compress, BinarySerialize does not automatically performs compression of the output. You pay the cost of a zlib compression only if needs be. Also some expressions like arrays (packed, raw) are already efficiently stored in memory, so having an output size of roughly the size of the byte count is generally a good result (Range produces packed array).

It is possible to get the contents of a HDF5 file as a ByteArray, it is one of the import elements. But I do not know what sorts of conversions it goes through to get there.