Issue 38913: Py_BuildValue("(s#O)", ...) segfaults if entered with exception raised

The following code results in a segmentation fault in CPython 3.8 while executes fines (raises a SystemError) in CPython 3.7.

PyObject* debug(PyObject *self, PyObject *args)
{
        const char * debug = "debug";
        PyErr_SetString(PyExc_ValueError, "debug!");
        return Py_BuildValue("(s#O)", debug, strlen(debug), Py_None);
}

It seems necessary for the format string to contain both an instance of "s#" and "O" to trigger the bug.

I'm attaching a C module that reproduces the problem.

seen with the current 3.8 branch.

Does it depend on whether PY_SSIZE_T_CLEAN is defined?

It is not reproducible with PY_SSIZE_T_CLEAN defined.

The problem arises from this code in do_mktuple(), staring at line 394 in modsupport.c:

            if (**p_format == '#') {
                ++*p_format;
                if (flags & FLAG_SIZE_T)
                    n = va_arg(*p_va, Py_ssize_t);
                else {
                    if (PyErr_WarnEx(PyExc_DeprecationWarning,
                                "PY_SSIZE_T_CLEAN will be required for '#' formats", 1)) {
                        return NULL;
                    }
                    n = va_arg(*p_va, int);
                }
            }

If this is entered with an exception raised, PyErr_WarnEx() return NULL, thus this function return NULL without consuming the argument relative to the string length for the "s#" specifier. This argument is then consumed at the next iteration for the "O" specifier, resulting in a segmentation fault when the string length is interpreted as an object pointer.

I don't know what is the best solution: either ignoring the return value of PyErr_WarnEx or swapping the lines from 

                    if (PyErr_WarnEx(PyExc_DeprecationWarning,
                                "PY_SSIZE_T_CLEAN will be required for '#' formats", 1)) {
                        return NULL;
                    }
                    n = va_arg(*p_va, int);


to 

                    n = va_arg(*p_va, int);
                    if (PyErr_WarnEx(PyExc_DeprecationWarning,
                                "PY_SSIZE_T_CLEAN will be required for '#' formats", 1)) {
                        return NULL;
                    }

The handling of the "y#" just below suffers from the same problem.

Note: this is going to miss Python 3.8.1 as I'm releasing 3.8.1rc1 right now.  Please address this before 3.8.2 (due in February).

Sadly, this missed the train for 3.8.2 as well.

New changeset 28d0bcac8b7e6dbd28311f1283dabb6a4d649fcb by Serhiy Storchaka in branch 'master':
bpo-38913: Fix segfault in Py_BuildValue("(s#O)", ...) if entered with exception raised. (GH-18656)
https://github.com/python/cpython/commit/28d0bcac8b7e6dbd28311f1283dabb6a4d649fcb

New changeset a7b8a969eb3daacb1fcb029a8c5fecb5d09c964b by Serhiy Storchaka in branch '3.8':
[3.8] bpo-38913: Fix segfault in Py_BuildValue("(sGH-O)", ...) if entered with exception raised. (GH-18656). (GH-18732)
https://github.com/python/cpython/commit/a7b8a969eb3daacb1fcb029a8c5fecb5d09c964b

After this change, the arm64 buildbots are reporting reference leaks:

1:03:24 load avg: 0.95 Re-running failed tests in verbose mode
1:03:24 load avg: 0.95 Re-running test_capi in verbose mode
test_capi leaked [4, 4, 4] references, sum=12

e.g. https://buildbot.python.org/all/#/builders/563/builds/25

Actually there is a leak in PyErr_WarnEx().

Opened issue39831.

nosy: + lukasz.langa, doko
messages: + msg357504

keywords: + 3.8regression