Page MenuHome

Keymap is not exported with the correct encoding
Closed, ResolvedPublicBUG

Description

System Information
Operating system: Windows-10-10.0.19041-SP0 64 Bits
Graphics card: GeForce RTX 2070/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 452.06

Blender Version
Broken: version: 2.90.1, branch: master, commit date: 2020-09-23 06:43, hash: rB3e85bb34d0d7
Worked: (newest version of Blender that worked as expected)

Short description of error
The exported keymap is not encoded with UTF-8 but an encoding depending on the locale. This produces a file which cannot be properly imported afterwards.

The keyconfig_export_as_data should specify the encoding when using open().

Exact steps for others to reproduce the error

  1. Change your locale to e.g. Simplified Chinese.
  2. Use the operator script to create an entry in the keymap.
  3. Change the string property of the operator in the keymap to Chinese characters.
  4. Export the keymap.

Event Timeline

Robert Guetzkow (rjg) changed the task status from Needs Triage to Needs Information from User.EditedNov 1 2020, 11:05 AM

The files you've attached appear to be encoded with GB2312 for simplified Chinese. Blender requires these files to be encoded with UTF-8. Additionally the keymap.py appears to be generally broken with NULL bytes at the beginning and seemingly corrupted output at the end of the file.

Could you please explain how you created these files and the exact steps to reproduce the issue? If Blender has saved these files that can't be imported, then this would be a bug. However, should you have manually edited the keymap and saved it with the unsupported encoding, then this is not a bug in Blender.

  1. I didn't do other extra operations, I clicked on export, then immediately click on import, then it appears this error, and the two errors are different, I don't know what else to pay attention to, could it be because I use the Chinese system to cause it,Also attached is a video of me in action.
  2. I've also found that creating a new keymap and selecting it also causes a bug!


我没有做其它多余的操作,我点击导出之后,立刻点击导入,然后它就出现了这样的报错,而且两个报错还不同,我也不知道还有其它什么地方注意的,会不会是因为我使用的中文系统导致的呢,另外附上我操作的视频
我还发现一个问题,就是新建一个 keymap 选择它也会出现BUG

Robert Guetzkow (rjg) added a comment.EditedNov 1 2020, 1:47 PM

@EMM (IMBI) Unfortunately the video in your last comment doesn't play for me. If you could re-upload it, that would be very helpful.

Just to avoid any misunderstandings, is it correct that you're doing the following steps?

  1. Open Blender
  2. Open the keymap user preferences Edit > Preferences > Keymap
  3. Select a keymap and then press Export (First error appears on export)
  4. Click on Import, select the previously exported file (Second error appears)

Was the first keymap that resulted in the export/import error created by editing one of the built-in keymaps through Edit > Preferences > Keymap? Is the EMM.py or .py (from your original report) the keymap what you get after exporting and you're trying to import this file again? Is keymap.py what you get after attempting to import EMM.py or .py?

I don't know what the "Blender-PME" addon is, but you use non-English names in it, I think so.

It could be something like this rBA86c4ca71ef18: Fix T76736: incorrect filename in SVG, fails to load elsewhere.

("wm.pme_user_pie_menu_call",
 {"type": 'SPACE', "value": 'PRESS'},
 {"properties":
  [("pie_menu_name", '±à¼­Ä£Ê½-Íø¸ñÊý¾Ý'),
   ("invoke_mode", 'HOTKEY'),
   ("keymap", '3D View'),
   ],
  },
 ),
EMM (IMBI) added a comment.EditedNov 1 2020, 2:26 PM
This is a re-uploaded video {F9151429} , but I can't seem to watch it either, so I've changed the .mp4 file to .py so you can download it, change the extension and you'll be able to view it!

这是重新上传的视频 {F9151429} ,但是我好像也看不了,所以我把这个.mp4文件改为.py,这样您就可以下载下来,更改一下后缀名称就能查看了

I don't know what the "Blender-PME" addon is, but you use non-English names in it, I think so.

Yes, I'm using a PIE plugin, it can use custom UI.
Its forum address is this:https://blenderartists.org/t/pie-menu-editor-1-18-0/662456/3680

是的,我正在使用一个PIE插件,它能使用自定义UI.是这个原因吗
它的论坛地址是这个:https://blenderartists.org/t/pie-menu-editor-1-18-0/662456/3680

Robert Guetzkow (rjg) added a comment.EditedNov 1 2020, 2:48 PM

I can reproduce the import error by attempting to import EMM.py. This is caused by the pie menu entries as @Yevgeny Makarov (jenkm) said and as I've noted in my first comment the encoding for these characters is not in UTF-8 but seem to be GB2312. It appears that the Blender-PME add-on doesn't handled encoding properly and stores non UTF-8 characters in the keymap. We can't provide support for the paid third-party add-on, however Blender shouldn't let an add-on store invalid characters in the keymap either.

I'll try to reproduce this issue through the Python API.

Edit: It's Blender's fault.

Robert Guetzkow (rjg) changed the task status from Needs Information from User to Needs Information from Developers.Nov 1 2020, 3:06 PM
Robert Guetzkow (rjg) added a comment.EditedNov 1 2020, 3:15 PM

@EMM (IMBI) As a workaround I've edited the EMM.py file which can now be imported. The Chinese characters have been replaced with the (poorly machine translated) English words and it's been saved with UTF-8 encoding.

Thank you, the file you sent can be imported, thank you very much, in addition to this bug I have feedback to the plug-in developers there, should be resolved!
谢谢您,您发的文件可以导入了,十分感谢,另外这个BUG我已经反馈到插件开发者那里了,应该会得到解决

I've tried to intentionally break the keymap through the Python API, but haven't been successful so far. Will continue to investigate.

Robert Guetzkow (rjg) added a comment.EditedNov 3 2020, 2:50 PM

It seems that the problem is caused when an operator has a string property, there is a shortcut registered for it in the keymap and the shortcut uses a value for the operators string property that is not UTF-8 encoded. However, I can't get a keymap to export after assigning e.g. Chinese characters to the property in the keymap preferences. It gives the following exception:

location: <unknown location>:-1
Traceback (most recent call last):
  File "Z:\02_software\blender-2.81a-windows64\2.81\scripts\startup\bl_operators\userpref.py", line 259, in execute
    all_keymaps=self.all,
  File "Z:\02_software\blender-2.81a-windows64\2.81\scripts\modules\bl_keymap_utils\io.py", line 194, in keyconfig_export_as_data
    fw(kmi_data)
  File "Z:\02_software\blender-2.81a-windows64\2.81\python\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 37-38: character maps to <undefined>

As of now I cannot reproduce the exact error from the original bug report with my current system configuration, but I have an idea. Since the error references cp1252.py Blender always seems to be using the system locale to determine the encoding. Reproducing the exact error from the original report may be possible with a different system locale configuration, which then presumably fails to import it later on. I would guess that in order to reproduce this bug you would have to set Windows' system locale to Simplified Chinese, which then results in code page 936 being used. This would then have GB2312 encoding applied to the input, which is what I've seen in the keymap included in the bug report.

If anyone wants to test this and investigate further, below is a script that implements a simple operator which is added to the keymap when registered.

Robert Guetzkow (rjg) added a comment.EditedNov 3 2020, 3:13 PM

I think I found the issue. The code page error in my last comment means that Blender isn't using UTF-8 for the export on my system either. The function keyconfig_export_as_data that writes the keymap doesn't set the encoding to UTF-8. The encoding should be explicitly set in Python's open() function.

Currently:

with open(filepath, "w") as fh:

Possible fix:

with open(filepath, "w", encoding="utf-8") as fh:

Edit: @Yevgeny Makarov (jenkm) was spot on with his comment.

Robert Guetzkow (rjg) added a comment.EditedNov 3 2020, 3:20 PM

@Yevgeny Makarov (jenkm) Yes, I know. Would've found/verified the problem quicker if I checked keyconfig_export_as_data or grepped open( first, instead of attempting to reproduce the reported error condition.

Robert Guetzkow (rjg) changed the task status from Needs Information from Developers to Confirmed.Nov 3 2020, 3:21 PM
Robert Guetzkow (rjg) renamed this task from Cannot import the same version of exported shortcuts 无法导入同一个版本导出的 快捷键 to Keymap is not exported with the correct encoding.Nov 3 2020, 3:24 PM
Robert Guetzkow (rjg) claimed this task.
Robert Guetzkow (rjg) updated the task description. (Show Details)
Robert Guetzkow (rjg) changed the subtype of this task from "Report" to "Bug".