Issue 35014: asyncio subprocess accepts string as parameter which lead to UnicodeEncodeError

Asyncio.create_subprocess_exec accepts a list of str as parameter which lead to UnicodeEncodeError I think it should accept only bytes shouldn't it?

List of strings works on both my local Linux box and CPython test suite.

Please provide more info about the error. Stacktrace can help

Hi Remy,

> Asyncio.create_subprocess_exec accepts a list of str as parameter which lead to UnicodeEncodeError I think it should accept only bytes shouldn't it?

Can you elaborate? On which OS? What is your error message? Can you paste a traceback?

> List of strings works on both my local Linux box and CPython test suite.

Indeed that's why I posted this bug report, in my opinion it should work only with bytes string.

> Can you elaborate? On which OS? What is your error message? Can you paste a traceback?

If you try to send a UTF-8 string on a linux box for instance, you might get a UnicodeEncodeError.

Let me try to provide you with a script to reproduce this error.

I though this would be sufficient to actually reproduce the issue.
However it seems that if the system encoding is UTF-8 it does work properly.

Here is the traceback I had:

```
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 69: ordinal not in range(128)
  File "worker.py", line 393, in <module>
    return_code = loop.run_until_complete(main(loop))
  File "asyncio/base_events.py", line 467, in run_until_complete
    return future.result()
  File "worker.py", line 346, in main
    '-f mp4', '-o', '{}/{}.mp4'.format(download_tempdir, video_id))
  File "worker.py", line 268, in run_command
    proc = await create
  File "asyncio/subprocess.py", line 225, in create_subprocess_exec
    stderr=stderr, **kwds)
  File "asyncio/base_events.py", line 1191, in subprocess_exec
    bufsize, **kwargs)
  File "asyncio/unix_events.py", line 191, in _make_subprocess_transport
    **kwargs)
  File "asyncio/base_subprocess.py", line 39, in __init__
    stderr=stderr, bufsize=bufsize, **kwargs)
  File "asyncio/unix_events.py", line 697, in _start
    universal_newlines=False, bufsize=bufsize, **kwargs)
  File "python3.6/subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "python3.6/subprocess.py", line 1267, in _execute_child
    restore_signals, start_new_session, preexec_fn)
```

I am adding the following info:

If I run the following on the Docker image where I got the error I get:

```
import sys
import locale

print(sys.getdefaultencoding())
print(locale.getpreferredencoding())
```

utf-8
ANSI_X3.4-1968

While if I run it on my machine I get:

utf-8
UTF-8

I don't know how to force the usage of the later locally to reproduce.

Settings LC_ALL=C and LANG=C didn't do the trick

Here we go:

```
$ python3.7 demo.py 
utf-8
UTF-8
Traceback (most recent call last):
  File "demo.py", line 21, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 568, in run_until_complete
    return future.result()
  File "demo.py", line 14, in main
    sys.stdout.write(out.decode('utf-8'))
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 1: ordinal not in range(128)
```

I think you'll get the same error on `subprocess.run()` call if your current locale is not UTF-8.

I don't recall the details but the Intenet has a lot info about setting locale per user and system-wide.

I believe Python 3.7 brings explicit unicode encoding/decoding.

If depending on the environment the create_subprocess_exec method can fail, I believe we should not try to encode the command lines attribute but rather enforce it to be bytes.

I added the UTF-8 Mode for you, for the Docker use case: python3.7 -X utf8. Using that, Python ignores your locale and speaks UTF-8.

What is your locale? Try the "locale" command.

Here are the locale set:

```
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
```

> LC_CTYPE="POSIX"

I modified Python 3.7.1 to enable the UTF-8 Mode when the LC_CTYPE is "POSIX". In Python 3.7.0, the UTF-8 Mode is only enabled if the LC_CTYPE is "C".

Unicode is complicated, the answer is somewhere here: https://unicodebook.readthedocs.io/

Sorry for the bothering, I thought it was a bug but apparently it's a feature. Thank you for your help, thank you for making Python better.

This issue is not an asyncio bug: the bug occurs in subprocess.

The bug is not a subprocess bug: subprocess works as expected, it encodes Unicode with sys.getfilesystemencoding() (see os.fsencode()).

The bug is that you use non-ASCII strings whereas your filesystem encoding is ASCII.

You have a different options to fix *your* issue:

* Use a different locale which uses a UTF-8 locale
* Enable the Python 3.7 UTF-8 mode
* Wait for Python 3.7.1 (which enables automatically the UTF-8 Mode for LC_CTYPE="POSIX")

Note: You might want to read my ebook http://unicodebook.readthedocs.io/ which explains how to deal with Unicode.

> I modified Python 3.7.1 to enable the UTF-8 Mode when the LC_CTYPE is "POSIX". In Python 3.7.0, the UTF-8 Mode is only enabled if the LC_CTYPE is "C"

Ok works for me thanks :)

stage: resolved

messages: + msg327962