Issue 18283: shutil.which() should support bytes

shutil.which() should support bytes. Some other functions in shutil module support bytes.

>>> shutil.which("echo")
'/bin/echo'                                                                                                                                                  
>>> shutil.which(b"echo")                                                                                                                                    
Traceback (most recent call last):                                                                                                                           
  File "<stdin>", line 1, in <module>                                                                                                                        
  File "/usr/lib64/python3.3/shutil.py", line 1126, in which
    name = os.path.join(dir, thefile)
  File "/usr/lib64/python3.3/posixpath.py", line 92, in join
    "components.") from None
TypeError: Can't mix strings and bytes in path components.
>>> shutil.which("echo", path="/bin")
'/bin/echo'
>>> shutil.which("echo", path=b"/bin")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.3/shutil.py", line 1098, in which
    path = path.split(os.pathsep)
TypeError: Type str doesn't support the buffer API
>>> shutil.which(b"echo", path=b"/bin")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.3/shutil.py", line 1098, in which
    path = path.split(os.pathsep)
TypeError: Type str doesn't support the buffer API

Here is an implementation.

What about bytearrays and other byte-like objects?

However I'm not sure this enhancement is worth to be accepted. Many other high-level functions in os and shutil modules do not support bytes paths. For shutil.which() there is no backward compatibility with Python 2 which we should support.

"What about bytearrays and other byte-like objects?

However I'm not sure this enhancement is worth to be accepted. Many other high-level functions in os and shutil modules do not support bytes paths. For shutil.which() there is no backward compatibility with Python 2 which we should support."

Bytes is the native type for path on all platforms... except Windows. So I fixed many functions of the os module to support bytes parameters. For example, functions of os.path support str and bytes types, but not bytearray. I also prefer to reject byte-like objects, to keep it simple.

The bytes type should be accepted for any kind of os function parameter: path, environment variable, command line argument, etc. It's not a question of being compatible with Python 2, an application can make the choice of only used bytes because bytes is the native type for OS data on UNIX.

Python 3 supports also "bytes stored in Unicode" thanks to the PEP 393, but it's not exactly the same: a path stored as str depends on the filesystem locale, whereas a bytes string is well defined (it's just an array of 8-bit unsigned numbers) and doesn't depend on any configuration variable.

All low-level functions in the os.path module supports string and bytes paths. But not all high-level functions in the shutil module supports bytes paths. Adding support for bytes paths will complicate implementation and tests.

Almost all functions of the shutil module support bytes filenames. I tested shutil.copyfile(), shutil.ignore_patterns(), shutil.copytree(), shutil.rmtree() and shutil.move().

I only found one function which only support Unicode filenames: shutil.make_archive().

New changeset a1a05e2724dd by Victor Stinner in branch 'default':
Issue #18283: shutil.which() now supports bytes argument, not only text argument.
http://hg.python.org/cpython/rev/a1a05e2724dd

which_bytes.patch does not work on Windows.

According to Serhiy, it's a new feature and so should wait for Python 3.5.

Would it be worthwhile to make a PR of this for 3.8?

> Would it be worthwhile to make a PR of this for 3.8?

Yes. Would you be interested to convert my old patch to a fresh PR? I can help you to work on that.

New changeset 5680f6546dcda550ad70eefa0a5ebf1375303307 by Victor Stinner (Cheryl Sabella) in branch 'master':
bpo-18283: Add support for bytes to shutil.which (GH-11818)
https://github.com/python/cpython/commit/5680f6546dcda550ad70eefa0a5ebf1375303307

messages: + msg335141
versions: + Python 3.8, - Python 3.5

versions: + Python 3.5, - Python 3.4