Issue 37076: _thread.start_new_thread(): call sys.unraisablehook() to handle uncaught exceptions

Python 3.8 now has 3 hooks for uncaught exceptions:

* sys.excepthook()
* sys.unraisablehook()
* threading.excepthook()

_thread.start_new_thread() calls none of these hooks, but directly logs the exception.

I propose to modify it to reuse sys.unraisablehook(): see attached PR.

--

Using threading.excepthook() would be another option, but _thread.start_new_thread() is usually wrapped in threading.Thread.run() which already uses threading.excepthook(). It might be surprising to see two bugs from the same thread using threading.excepthook().

Moreover, right now, _thread is the "low-level" API to access threads: it doesn't acces threading. I'm not comfortable by adding an inter-dependency between _thread (low-level) and threading (high-level).

If threading.Thread.run() is correctly written, it should not raise an exception. In the worst case, threading.excepthook should handle the exception. _thread.start_new_thread() should never get an exception raised by threading if threading.excepthook does correctly its job.

With PR 13617, the function which raised the exception can be retrieved using unraisable.object.

If we used threading.excepthook, I don't see how we could pass this function into threading.ExceptHookArgs: ExceptHookArgs.thread is expected to be a threading.Thread object.

This issue is a follow-up of:

* bpo-36829: Add sys.unraisablehook() to custom how "unraisable exceptions" are logged
* bpo-1230540: Add threading.excepthook() to handle uncaught exceptions raised by Thread.run()


Discussions around _thread.start_new_thead() exception:

Antoine Pitrou:
"""
From the doc: Handle uncaught :func:``Thread.run`` exception.

What if some C extension wants to report uncaught exceptions in C-created threads? Does it have to create a dummy Thread object?

For example, how about threads created by _thread.start_new_thread?
"""

https://github.com/python/cpython/pull/13515#issuecomment-495935350


Serhiy Storchaka:
"If we want to support low-level threads created by start_new_thread() we should call [threading.]excepthook() from t_bootstrap instead of Thread._bootstrap_inner. Otherwise implementing excepthook() as a Thread method would be more convenient."

https://bugs.python.org/issue1230540#msg343748

> _thread.start_new_thread() calls none of these hooks, but directly logs the exception.

It calls sys.excepthook() currently:

import _thread
import threading
import sys

done = False
def hook(*args):
    global done
    print(threading.current_thread())
    done = True
sys.excepthook = hook

def worker():
    raise Exception
_thread.start_new_thread(worker, tuple())
while not done:
    pass

Output:
---
Unhandled exception in thread started by <function worker at 0x7fae27f212d0>
<_DummyThread(Dummy-1, started daemon 140385971111680)>
---

Ah right, sys.excepthook is called! But "Unhandled exception in thread started by <function worker at 0x7fae27f212d0>" line is always written into stderr, it cannot be avoided :-( And sys.excepthook doesn't provide access to the thread function which caused the issue.

I updated my PR description:

_thread.start_new_thread() now logs uncaught exception raised by the
function using sys.unraisablehook(), rather than sys.excepthook(), so
the hook gets access to the function which raised the exception.

> If threading.Thread.run() is correctly written, it should not raise an exception.

Since the error handling for threading.Thread.run() is written on Python there are many opportunities to get an exception in error handling: KeyboardInterrupt, MemoryError and, at the shutdown stage, maybe NameError, AttributeError or TypeError.

> Since the error handling for threading.Thread.run() is written on Python there are many opportunities to get an exception in error handling: KeyboardInterrupt, MemoryError and, at the shutdown stage, maybe NameError, AttributeError or TypeError.

Sure, the risk is real, but I tried to minimize it.

NameError and AttributeError "should" not happen: I wrote _make_invoke_excepthook() to prevent name errors. Functions used to invoke threading.excepthook are "cached" in a private namespace. Moreover, the default hook is implemented in C also to reduce the risk of raising a new exception.

Anyway, if threading.excepthook() raises a second exception, sys.excepthook() is now called to handle it ;-) That's a Python 3.8 change. In Python 3.7 and older, new exception was handled by start_new_thread().

New changeset 8b09500345d998f3ff1e363a5210bc87f42ff306 by Victor Stinner in branch 'master':
bpo-37076: _thread.start_new_thread() calls _PyErr_WriteUnraisableMsg() (GH-13617)
https://github.com/python/cpython/commit/8b09500345d998f3ff1e363a5210bc87f42ff306

Thanks Serhiy for the review!

stage: patch review -> resolved