Not my area either but what you call unicode_array is essentially UTF32, so naming isn't great and leaves a rather large amount of ambiguity what type these variables actually are.

i'd rather see BLI_strncpy_utf32_as_utf8 rather than BLI_unicode_array_as_str_utf8 and BLI_UTF32 rather than BLI_unicode

but i'll let @Campbell Barton (campbellbarton) to be the deciding factor here. so hold off on doing any changes based on my feedback.

Agree the API should be explicit WRT utf8,16,32... etc.

Since C & C++ have char32_t, we could use this. If it's not defined in some cases it could be typedef'd (probably in BLI_compiler_compat.h, or one of the BLI_string headers).

Examples to uses in C/C++.

• https://docs.microsoft.com/en-us/cpp/cpp/char-wchar-t-char16-t-char32-t?view=vs-2019
• https://ld2016.scusa.lsu.edu/cppreference/en/c/string/multibyte.html

If this introduces too many problems supporting different compilers/libc versions, we could define BLI_char32_t which can be replaced by char32_t later, when compiler support has improved.

For API naming:

• BLI_str_utf8_as_utf32 (was BLI_str_utf8_as_unicode_array)
• BLI_str_utf32_as_utf8 (was BLI_unicode_array_as_str_utf8)
• BLI_str_utf32_as_utf8_len (was BLI_unicode_array_utf8_len)

Some existing uses of unicode should be renamed to utf32 as well.

Couple of things that stick out:

1. there's a bunch of functions with wchar in their name that take a char_32t type, a better name will have to be picked for those.
2. for memory allocations sizeof(wchar_t) is still used here and there in the char32_t codepaths which with allocate a too small buffer as a result.

Only checked this on Linux, if this works on other platforms - LGTM.

Note that some further refactoring would be good after this commit, eg:

BLI_str_utf8_as_utf32 and BLI_str_utf8_as_unicode are confusing, BLI_str_utf8_as_unicode could be renamed to BLI_str_utf8_as_utf32_codepoint, but that can be handled separately.