Issue 36465: Make release and debug ABI compatible

When Python is built in debug mode, PyObject gets 2 new fields: _ob_prev and _ob_next. These fields change the offset of following fields in the PyObject structure and so breaks the ABI.

I propose to modify the debug build (Py_DEBUG) to not imply Py_TRACE_REFS anymore. Antoine Pitrou proposed this idea when the C API was discussed to get a stable ABI.

Another more radical idea is to completely remove Py_TRACE_REFS special build.

> Another more radical idea is to completely remove Py_TRACE_REFS special build.

I wrote PR 12614 to implement this idea. I wrote it to see which code depends on it:

commit 63509498761a0e7f72585a8cd7df325ea2abd1b2 (HEAD -> remove_trace_refs, origin/remove_trace_refs)
Author: Victor Stinner <vstinner@redhat.com>
Date:   Thu Mar 28 23:26:58 2019 +0100

    WIP: bpo-36465: Remove Py_TRACE_REFS special build
    
    Remove _ob_prev and _ob_next fields of PyObject when Python is
    compiled in debug mode to make debug ABI closer to the release ABI.
    
    Remove:
    
    * sys.getobjects()
    * PYTHONDUMPREFS environment variable
    * _PyCoreConfig.dump_refs
    * PyObject._ob_prev and PyObject._ob_next fields
    * _PyObject_HEAD_EXTRA and _PyObject_EXTRA_INIT macros
    * _Py_AddToAllObjects()
    * _Py_PrintReferenceAddresses()
    * _Py_PrintReferences()
    * _Py_ForgetReference(op) is replaced with _Py_INC_TPFREES(op)

I never used PYTHONDUMPREFS. I just tried in Python 3.7: Python does crash with this option... so this option doesn't sound popuplar. Otherwise, many users would complain.

$ PYTHONDUMPREFS=1 python3.7-dbg -c pass
...
0x7f7eae14aa90 [1] 'Thread-local dummy'
0x7f7eae19b448 [1] (<class '_thread._localdummy'>, <class 'object'>)
0x7f7eae14aa30 [1] {'__doc__': 'Thread-local dummy'}
0x7f7eae1356d8 [1] (<class 'object'>,)
0x7d0940 [2] <class '_thread._localdummy'>
0x7f7eae120d58 [1] Segmentation fault (core dumped)

I never used sys.getobjects() neither, but I can imagine that someone might want to use for very specific needs. So maybe it's safer to not immediately remove the feature. At least, a deprecation period would be needed.

I suggest to reject PR 12614 and not remove Py_TRACE_REFS.

PR 12615 changes Py_DEBUG to no longer imply Py_TRACE_REFS. IMHO it's a more reasonable approach.

I'm not sure if it's enough to magically get exactly the same ABI than Python built in release mode.

Since my goal here is to support a C extension compiled in release mode on a Python compiled in debug mode, Py_TRACE_REFS should be a new separated ABI since it modify Py_INCREF/Py_DECREF. Maybe a new configure option should be added to opt-in for Py_TRACE_REFS and add "t" (T stands for Trace references) to SOABI.

I started a thread on python-dev to discuss this issue:
"[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode"
https://mail.python.org/pipermail/python-dev/2019-April/157178.html

It's a follow-up of my previous thread:
"[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build"
https://mail.python.org/pipermail/python-dev/2019-April/157015.html

New changeset f4e4703e746067d6630410408d414b11003334d6 by Victor Stinner in branch 'master':
bpo-36465: Make release and debug ABI compatible (GH-12615)
https://github.com/python/cpython/commit/f4e4703e746067d6630410408d414b11003334d6

New changeset 235e7b2b0d937ba8b4a9172aa72206187e3e1f54 by Victor Stinner in branch 'master':
bpo-36465: Fix test_regrtest on Windows (GH-12945)
https://github.com/python/cpython/commit/235e7b2b0d937ba8b4a9172aa72206187e3e1f54

New changeset a7847590f07655e794d7c62130aea245a110acef by Hai Shi in branch 'master':
bpo-36465: Update doc of init_config.rst (GH-18520)
https://github.com/python/cpython/commit/a7847590f07655e794d7c62130aea245a110acef