Fix T74510 wchar_t losing 16 bits of data on windows.
ClosedPublic
Actions

Authored by Ray Molenkamp (LazyDodo) on Dec 10 2020, 9:51 PM.

Details

Reviewers

Maniphest Tasks

T74510: Various Addons using filesystem properties (PROP_FILEPATH etc): UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe0 in position 24: invalid continuation byte

Commits

rB5496d8cd3613: Windows: Fix wchar_t truncation

Summary

This fixes the issues seen in T74510 the root cause of this issue is BLI_strncpy_wchar_from_utf8 assuming wchar_t being 32 bits for windows where it is not (UTF16 there), so it just clips off the upper 16 bits and asserts in case that ever happened. Which naturally came to bite us in the rear with some international users.

Given most of the code in BLI_strncpy_wchar_from_utf8 was a duplicate of BLI_str_utf8_as_utf32 this patch just calls the right functions for the right platforms.

Tests pass on windows, have not tested on linux/mac

Diff Detail

Repository: rB Blender

Event Timeline

Ray Molenkamp (LazyDodo) requested review of this revision.Dec 10 2020, 9:51 PM

Ray Molenkamp (LazyDodo) created this revision.

Harbormaster completed remote builds in B11679: Diff 31840.Dec 10 2020, 9:51 PM

Ray Molenkamp (LazyDodo) retitled this revision from Fix T74510 to Fix T74510 wchar_t losing 16 bits of data on windows. .Dec 10 2020, 9:52 PM

Ray Molenkamp (LazyDodo) edited the summary of this revision. (Show Details)

Robert Guetzkow (rjg) added a subscriber: Robert Guetzkow (rjg).Dec 10 2020, 9:58 PM

Raimund Klink (Raimund58) awarded a token.Dec 10 2020, 10:51 PM

Raimund Klink (Raimund58) added a subscriber: Raimund Klink (Raimund58).

It would be great if this would make it into the 2.91.1 release :D
(We would voluntary test this build ;) )

Robert Guetzkow (rjg) awarded a token.Dec 10 2020, 11:00 PM

Robert Guetzkow (rjg) added inline comments.Dec 10 2020, 11:11 PM

source/blender/blenlib/intern/string_utf8.c
398–401	This should be `ifdef WIN32` otherwise this generates a warning on Linux as `WIN32` is not defined.

Robert Guetzkow (rjg) added inline comments.Dec 10 2020, 11:22 PM

source/blender/blenlib/intern/string_utf8.c
401	GCC warns that the signedness of `wchar_t` (signed) is not the same as `char32_t` (unsigned).

like i said, i'm unable to test on linux/mac, i suppose a cast between wchar32_t and wchar_t would be permissible here?

A cast to (char32_t *) (without a w, see signature of BLI_str_utf8_as_utf32) should be fine, since this is a signed to unsigned cast. I haven't tested if it works properly on Linux yet, but will do tomorrow.