[proxy] web.archive.org← back | site home | direct (HTTPS) ↗ | proxy home | ◑ dark◐ light

Issue 32370: Wrong ANSI encoding used by subprocess for some locales

Created on 2017-12-19 00:24 by Segev Finer, last changed 2018-02-13 09:18 by serhiy.storchaka. This issue is now closed.

Messages (6) msg308597 - (view) Author: Segev Finer (Segev Finer) * Date: 2017-12-19 00:24
The following test is failing randomly for me (python -m test test_uuid -v):
======================================================================
ERROR: test_ipconfig_getnode (test.test_uuid.TestInternalsWithoutExtModule)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "cpython\lib\test\test_uuid.py", line 551, in test_ipconfig_getnode
    node = self.uuid._ipconfig_getnode()
  File "cpython\lib\uuid.py", line 487, in _ipconfig_getnode
    for line in pipe:
  File "cpython\lib\encodings\cp1255.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 682: character maps to <undefined>

This is caused by trying to decode the output of "ipconfig" using cp1255, while the output really uses cp862 and annoyingly it started to print times using the current locale (Which displays broken in the console anyhow, question mark boxes... *sigh*) in some Windows version (Using Windows 10.0.16299 ATM).
msg308614 - (view) Author: Eryk Sun (eryksun) * Date: 2017-12-19 05:44
ipconfig uses (or defaults to) OEM encoded output when writing to a pipe or file. On the other hand, Python's TextIOWrapper defaults to ANSI (i.e. 'mbcs'). In 3.6+, uuid._ipconfig_getnode could be rewritten to call subprocess.Popen with the new 'oem' encoding. 

In Windows 10, the networking command-line utilities (e.g. ipconfig, netstat, nbtstat, ping, tracert, hostname, finger, and ftp) support an "OutputEncoding" environment variable. Its value can be set to "Unicode" (UTF-16), "UTF-8", or "Ansi". I don't think this is supported in Windows 7, however.
msg311952 - (view) Author: Steve Dower (steve.dower) * Date: 2018-02-10 16:01
The patch looks good, but the rest of the code for this looks horrible(and potentially a security risk)... maybe we ought to take this fix for now but prioritise replacing this whole function with a C implementation or at least a better search for the ipconfig tool.
msg312104 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * Date: 2018-02-13 06:29
New changeset da6c3da6c33c6bf794f741e348b9c6d86cc43ec5 by Serhiy Storchaka (Segev Finer) in branch 'master':
bpo-32370: Use the correct encoding for ipconfig output in the uuid module. (GH-5608)
https://github.com/python/cpython/commit/da6c3da6c33c6bf794f741e348b9c6d86cc43ec5
msg312110 - (view) Author: miss-islington (miss-islington) Date: 2018-02-13 08:58
New changeset 9b5a90b975ef32b261d60b8ec06504f4ffd00d63 by Miss Islington (bot) in branch '3.7':
bpo-32370: Use the correct encoding for ipconfig output in the uuid module. (GH-5608)
https://github.com/python/cpython/commit/9b5a90b975ef32b261d60b8ec06504f4ffd00d63
msg312111 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * Date: 2018-02-13 09:15
New changeset c3f9d7e0ea30e94c901d13e1d43ff0be1e5dbcb7 by Serhiy Storchaka in branch '3.6':
[3.6] bpo-32370: Use the correct encoding for ipconfig output in the uuid module. (GH-5608). (#5654)
https://github.com/python/cpython/commit/c3f9d7e0ea30e94c901d13e1d43ff0be1e5dbcb7
History Date User Action Args 2018-02-13 09:18:06serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved 2018-02-13 09:15:26serhiy.storchakasetmessages: + msg312111 2018-02-13 08:58:30miss-islingtonsetnosy: + miss-islington
messages: + msg312110
2018-02-13 07:24:44serhiy.storchakasetpull_requests: + pull_request5456 2018-02-13 06:31:02miss-islingtonsetpull_requests: + pull_request5454 2018-02-13 06:29:57serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg312104
2018-02-11 10:48:12serhiy.storchakasetversions: + Python 3.6 2018-02-10 16:01:36steve.dowersetmessages: + msg311952 2018-02-10 13:43:45Segev Finersetversions: + Python 3.8 2018-02-10 13:40:58Segev Finersetkeywords: + patch
stage: patch review
pull_requests: + pull_request5418 2017-12-19 05:44:09eryksunsetnosy: + eryksun
messages: + msg308614
2017-12-19 00:24:51Segev Finercreate