Issue 19865: create_unicode_buffer() fails on non-BMP strings on Windows

create_unicode_buffer() fails on Windows if the initializer string contains unicode code points outside of the Basic Multilingual Plane and an explicit length is not specified.

The problem appears to be rooted in the fact that, since PEP 393, len() returns the number of code points, which does not always correspond to the number of 16-bit wchar words needed for the encoding on Windows. Because of that, the preallocated c_wchar buffer will be too short for the UTF-16 string.

The following small snippet demonstrates the problem:

from ctypes import create_unicode_buffer
b = create_unicode_buffer("\U00028318\U00028319")
print(b)

  File "c:\Python33\lib\ctypes\__init__.py", line 294, in create_unicode_buffer
    buf.value = init
ValueError: string too long

I can confirm that this problem still exists so can someone take a look please, thanks.

When sizeof(c_wchar) == 2, it can just count the number of non-BMP ordinals in the string. Another approach would be to use size = pythonapi.PyUnicode_AsWideChar(init, None, 0), but then the whole function may as well be implemented in the _ctypes extension module.

I'm still able to reproduce this issue with ctypes under Python 3.7.0

I have created a pull request for this issue. Please take a look.

New changeset 9765efcb39fc03d5b1abec3924388974470a8bd5 by Victor Stinner (Zackery Spytz) in branch 'master':
bpo-19865: ctypes.create_unicode_buffer() supports non-BMP strings on Windows (GH-14081)
https://github.com/python/cpython/commit/9765efcb39fc03d5b1abec3924388974470a8bd5

New changeset 0b592d513b073cd3a4ba7632907c25b8282f15ce by Miss Islington (bot) in branch '3.7':
bpo-19865: ctypes.create_unicode_buffer() supports non-BMP strings on Windows (GH-14081)
https://github.com/python/cpython/commit/0b592d513b073cd3a4ba7632907c25b8282f15ce

New changeset b0f6fa8d7d4c6d8263094124df9ef9cf816bbed6 by Miss Islington (bot) in branch '3.8':
bpo-19865: ctypes.create_unicode_buffer() supports non-BMP strings on Windows (GH-14081)
https://github.com/python/cpython/commit/b0f6fa8d7d4c6d8263094124df9ef9cf816bbed6

Thanks Zackery Spytz for the fix. Thanks Gergely Erdélyi for the bug report! Sorry for the long delay.

stage: patch review -> resolved

messages: + msg345596
versions: + Python 3.9, - Python 3.4, Python 3.5, Python 3.6

messages: + msg228405
versions: + Python 3.5, - Python 3.3