Issue 21932: os.read() must use Py_ssize_t for the size parameter

os.read() currently uses the int type for the size parameter, whereas the C function uses a C size_t parameter:

       ssize_t read(int fd, void *buf, size_t count);

The Python function must use the Py_ssize_t type, but truncate to INT_MAX on Windows, as done in FileIO.read(). It would help to implement FileIO in pure Python: issue #21859.

I don't think that this enhancement should be backported to Python 2.7 nor Python 3.4.

Patch LGTM. Here is a test for it.

New changeset e9b401d46e20 by Victor Stinner in branch 'default':
Issue #21932: os.read() now uses a :c:func:`Py_ssize_t` type instead of
http://hg.python.org/cpython/rev/e9b401d46e20

> Patch LGTM.

Thanks for the review.

> Here is a test for it.

Your test does not pass because Linux truncates the read() size to 2GB - 4096. I tested with /dev/zero device and with a regular file.

I wrote a simplified test to just check that size larger than INT_MAX does not emit an OverflowError.

New changeset 4a7fcd5273ce by Victor Stinner in branch 'default':
Issue #21932: Ooops, os.read(fd, size) allocates a buffer of size bytes, even
http://hg.python.org/cpython/rev/4a7fcd5273ce

Failed on 32-bit:

======================================================================
ERROR: test_large_read (test.test_os.FileTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/support/__init__.py", line 1592, in wrapper
    return f(self, maxsize)
  File "/home/serhiy/py/cpython/Lib/test/test_os.py", line 136, in test_large_read
    data = os.read(fp.fileno(), size)
OverflowError: Python int too large to convert to C ssize_t

----------------------------------------------------------------------

Re-add the @unittest.skipUnless(INT_MAX < PY_SSIZE_T_MAX, "needs INT_MAX < PY_SSIZE_T_MAX") decorator (between cpython_only and bigmemtest).

New changeset 880e2cdac8b1 by Victor Stinner in branch 'default':
Issue #21932: Skip test_os.test_large_read() on 32-bit system
http://hg.python.org/cpython/rev/880e2cdac8b1

> Re-add the @unittest.skipUnless(INT_MAX < PY_SSIZE_T_MAX, "needs INT_MAX < PY_SSIZE_T_MAX") decorator (between cpython_only and bigmemtest).

Oh, I removed it because I thought that it was useless. I didn't understand the purpose of the test. I added a comment. Thanks for the report.

Thank you for fixing os.read().

> Serhiy Storchaka added the comment:
>
> Thank you for fixing os.read().
>

You're welcome, thanks for your help with the test. I hope that it would
help you to implement FileIO in Python.

FYI I saw EINVAL errors on test_large_read() on some buildbots, Mac OS X
and FreeBSD if I remember correctly, when I forgot the bigmem decorator.
And FreeBSD kills the process if there is not enough memory.

messages: + msg222465