Issue 36974: Implement PEP 590

New changeset aacc77fbd77640a8f03638216fa09372cc21673d by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: implement PEP 590 (GH-13185)
https://github.com/python/cpython/commit/aacc77fbd77640a8f03638216fa09372cc21673d

New changeset fecb75c1bb46c818e6579ba422cfa5d0d9d104d1 by Petr Viktorin in branch 'master':
bpo-36974: Fix GDB integration (GH-13665)
https://github.com/python/cpython/commit/fecb75c1bb46c818e6579ba422cfa5d0d9d104d1

BUILDBOT FAILURE REPORT
=======================

Builder name: AMD64 Ubuntu Shared 3.x
Builder url: https://buildbot.python.org/all/#/builders/141/
Build url: https://buildbot.python.org/all/#/builders/141/builds/1866

Failed tests
------------

- test_pycfunction (test.test_gdb.PyBtTests)


Test leaking resources
----------------------



Build summary
-------------

== Tests result: FAILURE then FAILURE ==

405 tests OK.

10 slowest tests:
- test_multiprocessing_spawn: 6 min 27 sec
- test_tools: 5 min 34 sec
- test_concurrent_futures: 5 min 13 sec
- test_tokenize: 4 min 43 sec
- test_lib2to3: 3 min 58 sec
- test_gdb: 3 min 16 sec
- test_multiprocessing_forkserver: 2 min 25 sec
- test_asyncio: 2 min 5 sec
- test_multiprocessing_fork: 1 min 40 sec
- test_capi: 1 min 36 sec

1 test failed:
    test_gdb

17 tests skipped:
    test_devpoll test_idle test_ioctl test_kqueue test_msilib
    test_ossaudiodev test_startfile test_tcl test_tix test_tk
    test_ttk_guionly test_ttk_textonly test_turtle test_winconsoleio
    test_winreg test_winsound test_zipfile64

1 re-run test:
    test_gdb

Total duration: 42 min 45 sec


Tracebacks
----------

```traceback
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_gdb.py", line 890, in test_pycfunction
    self.assertIn('#2 <built-in method gmtime', gdb_output)
AssertionError: '#2 <built-in method gmtime' not found in 'Breakpoint 1 (time_gmtime) pending.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".\n\nBreakpoint 1, time_gmtime (self=<module at remote 0x7ffff7f05fb0>, args=(1,)) at ./Modules/timemodule.c:446\n446\t{\n#6 Frame 0x7ffff7f11a50, for file <string>, line 3, in foo ()\n#12 Frame 0x5555557be620, for file <string>, line 5, in bar ()\n#18 Frame 0x5555557be3f0, for file <string>, line 6, in <module> ()\n'


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_gdb.py", line 890, in test_pycfunction
    self.assertIn('#2 <built-in method gmtime', gdb_output)
AssertionError: '#2 <built-in method gmtime' not found in 'Breakpoint 1 (time_gmtime) pending.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".\n\nBreakpoint 1, time_gmtime (self=<module at remote 0x7ffff7f05fb0>, args=(1,)) at ./Modules/timemodule.c:446\n446\t{\n#6 Frame 0x7ffff7f11a50, for file <string>, line 3, in foo ()\n#12 Frame 0x5555557be590, for file <string>, line 5, in bar ()\n#18 Frame 0x5555557be360, for file <string>, line 6, in <module> ()\n'

```

Current builder status
----------------------

The builder is failing currently

Commits
-------

- 0c2f9305640f7655ba0cd5f478948b2763b376b3

- aacc77fbd77640a8f03638216fa09372cc21673d


Other builds with similar failures
----------------------------------

-  https://buildbot.python.org/all/#/builders/21/builds/3076
-  https://buildbot.python.org/all/#/builders/21/builds/3077
-  https://buildbot.python.org/all/#/builders/21/builds/3078
-  https://buildbot.python.org/all/#/builders/21/builds/3079
-  https://buildbot.python.org/all/#/builders/21/builds/3080
-  https://buildbot.python.org/all/#/builders/13/builds/3087
-  https://buildbot.python.org/all/#/builders/13/builds/3088
-  https://buildbot.python.org/all/#/builders/13/builds/3089
-  https://buildbot.python.org/all/#/builders/13/builds/3090
-  https://buildbot.python.org/all/#/builders/85/builds/2883
-  https://buildbot.python.org/all/#/builders/85/builds/2884
-  https://buildbot.python.org/all/#/builders/85/builds/2885
-  https://buildbot.python.org/all/#/builders/85/builds/2886
-  https://buildbot.python.org/all/#/builders/141/builds/1869
-  https://buildbot.python.org/all/#/builders/176/builds/590
-  https://buildbot.python.org/all/#/builders/176/builds/591
-  https://buildbot.python.org/all/#/builders/176/builds/592
-  https://buildbot.python.org/all/#/builders/176/builds/593
-  https://buildbot.python.org/all/#/builders/176/builds/594
-  https://buildbot.python.org/all/#/builders/16/builds/3055
-  https://buildbot.python.org/all/#/builders/16/builds/3056
-  https://buildbot.python.org/all/#/builders/16/builds/3057
-  https://buildbot.python.org/all/#/builders/16/builds/3058
-  https://buildbot.python.org/all/#/builders/16/builds/3059

Common commits for all builds:

- aacc77fbd77640a8f03638216fa09372cc21673d

@Petr is https://bugs.python.org/issue37090 and https://github.com/python/cpython/pull/13668 also addressing the buildbot failures?

No, just https://github.com/python/cpython/pull/13665 is addressing the  failures. And it seems that it's successful -- only the slowest of the failed ones (AMD64 Ubuntu Shared 3.x) is still red.

bpo-37090 extends the test so it can catch more bugs in the future (but there's no rush to get it in...)

Great! Thank you very much for the quick fix for the problem.

For AMD64 Ubuntu Shared 3.x, the last build was successful:

https://buildbot.python.org/all/#/builders/141/builds/1870/steps/5/logs/stdio

All stable buildbots are back to green.

New changeset 735e8afa9ee942367b5d0807633a2b9f662cbdbf by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: inherit the vectorcall protocol (GH-13498)
https://github.com/python/cpython/commit/735e8afa9ee942367b5d0807633a2b9f662cbdbf

New changeset c145f3bfbe80d498d40848450d4d33c14e2cf782 by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: remove _PyObject_HasFastCall (GH-13460)
https://github.com/python/cpython/commit/c145f3bfbe80d498d40848450d4d33c14e2cf782

New changeset 37788bc23f6f1ed0362b9b3b248daf296c024849 by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: rename _FastCallKeywords -> _Vectorcall (GH-13653)
https://github.com/python/cpython/commit/37788bc23f6f1ed0362b9b3b248daf296c024849

New changeset 530f506ac91338b55cf2be71b1cdf50cb077512f by Benjamin Peterson (Jeroen Demeyer) in branch 'master':
bpo-36974: tp_print -> tp_vectorcall_offset and tp_reserved -> tp_as_async (GH-13464)
https://github.com/python/cpython/commit/530f506ac91338b55cf2be71b1cdf50cb077512f

I found an issue in PEP 590:

When inheriting a heap subclass from a vectorcall class that sets .tp_call=PyVectorcall_Call (as recommended), the subclass does not inherit _Py_TPFLAGS_HAVE_VECTORCALL, and thus PyVectorcall_Call does not work for it.
Possible solutions come to mind:
- Inherit tp_vectorcall_offset more normally but handle setting __call__ specially
- Inherit tp_vectorcall_offset (but not _Py_TPFLAGS_HAVE_VECTORCALL) more normally, and make PyVectorcall_Call ignore _Py_TPFLAGS_HAVE_VECTORCALL

I'll send a PR for the latter.

New changeset fb9423fd0a85f06affb8c3a8f25dd598a649aa42 by Petr Viktorin in branch 'master':
bpo-36974: Make tp_call=PyVectorcall_Call work for inherited types (GH-13699)
https://github.com/python/cpython/commit/fb9423fd0a85f06affb8c3a8f25dd598a649aa42

New changeset e584cbff1ea78e700cf9943d50467e3b58301ccc by Petr Viktorin in branch 'master':
bpo-36027 bpo-36974: Fix "incompatible pointer type" compiler warnings (GH-13758)
https://github.com/python/cpython/commit/e584cbff1ea78e700cf9943d50467e3b58301ccc

New changeset 9e3e06e582accec82eb29cf665c3b4c7d84d2eb0 by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: document PEP 590 (GH-13450)
https://github.com/python/cpython/commit/9e3e06e582accec82eb29cf665c3b4c7d84d2eb0

New changeset be718c33f06b3496faa61142df24fb071fd5d1f1 by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: add some assertions for PEP 590 (GH-13682)
https://github.com/python/cpython/commit/be718c33f06b3496faa61142df24fb071fd5d1f1

New changeset a8b27e623d75377aabe50df27e97cab4e81a174a by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: inherit tp_vectorcall_offset unconditionally (GH-13858)
https://github.com/python/cpython/commit/a8b27e623d75377aabe50df27e97cab4e81a174a

New changeset 26fe6c35374fa32577b230b856a92a3b094e08ed by Petr Viktorin (Miss Islington (bot)) in branch '3.8':
bpo-36974: inherit tp_vectorcall_offset unconditionally (GH-13858) (GH-14342)
https://github.com/python/cpython/commit/26fe6c35374fa32577b230b856a92a3b094e08ed

New changeset 0d722f3cd602e5f5492f9c65c8af57ea9d3743b6 by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: separate vectorcall functions for each calling convention (GH-13781)
https://github.com/python/cpython/commit/0d722f3cd602e5f5492f9c65c8af57ea9d3743b6

Any objections to closing this?

Buildbots are green, closing.

Thank you for the implementation!

New changeset bf8e82f976b37856c7d35cdf88a238cb6f57fe65 by Łukasz Langa (Jeroen Demeyer) in branch '3.8':
[3.8] bpo-36974: separate vectorcall functions for each calling convention (GH-13781) (#14782)
https://github.com/python/cpython/commit/bf8e82f976b37856c7d35cdf88a238cb6f57fe65

> [3.8] bpo-36974: separate vectorcall functions for each calling convention (GH-13781) (#14782)
> https://github.com/python/cpython/commit/bf8e82f976b37856c7d35cdf88a238cb6f57fe65

FYI this change caused a regression in libcomps with Python 3.8 beta3, whereas it works well with Python 3.8 beta2.

It's not a bug in Python, but it was a bug in libcomps (already fixed upstream). I just fixed libcomps:
https://github.com/rpm-software-management/libcomps/pull/50

This project used the following method descriptors (for module functions):

{"categories_match", (PyCFunction)PyCOMPS_categories_match, METH_KEYWORDS,
 PyCOMPS_validate__doc__},
{"environments_match", (PyCFunction)PyCOMPS_envs_match, METH_KEYWORDS,
 PyCOMPS_validate__doc__},

In Python 3.7, importing the module was just fine: descriptor flags are only checked at the  first call to the method.

In Python 3.8, descriptor flags are checked when the module is imported.

Fedora bug: https://bugzilla.redhat.com/show_bug.cgi?id=1734777

The fix is to use the right flags: "METH_VARARGS | METH_KEYWORDS" instead of "METH_KEYWORDS".

Should we add a note like "if you get a 'SystemError: bad call flags' on import, check the descriptor flags of your functions" in What's New in Python 3.8? Maybe with a link to this issue.
https://docs.python.org/dev/whatsnew/3.8.html#changes-in-the-c-api

> Should we add a note like "if you get a 'SystemError: bad call flags' on import, check the descriptor flags of your functions" in What's New in Python 3.8?

A better solution would be to change the error message. We could change it to something like SystemError("bad flags 0x2 for PyCOMPS_categories_match")

But maybe it's not worth it. Are there many projects that define functions/methods but never call them?

'SystemError: bad call flags' error message LGTM, I don't think that it should be changed.

> Are there many projects that define functions/methods but never call them?

I have no idea.

New changeset 9a13a388f202268dd7b771638adbec132449b98b by Miss Islington (bot) (Jeroen Demeyer) in branch 'master':
bpo-36974: expand call protocol documentation (GH-13844)
https://github.com/python/cpython/commit/9a13a388f202268dd7b771638adbec132449b98b

issue39245 tracks making the API public in Python 3.9.

I created bpo-39361: [C API] Document PyTypeObject.tp_print removal in What's New In Python 3.9.

stage: patch review -> resolved