On Windows, if a worker is terminated too quickly, handles created by DupHandle() on reduction.dump() remain open in the parent process, causing a handles leak. Pool now waits using events until all new workers started to make sure that the workers closed handles inherited by DupHandle.
|
I tried a different approach avoiding DUPLICATE_CLOSE_SOURCE in PR #7965, but it caused test_wait() to hang and it changed the semantics (handles are no longer closed in the parent as soon as they are "stolen" by the child process). |
|
@pitrou: It seems like this bugfix is the last remaining issue to unblock my PR 7827 (bpo-18174: regrtest -R 3:3 checks for handle leak) to detect Windows handles leaks. Would you mind to review it? Note: I chose to be conservative and set the event late, just before handling the first task in the worker. If we only want to fix https://bugs.python.org/issue18174 the event could be set earlier, just at the entry of the worker() function. |
|
The problem is this will make the >>> ctx = multiprocessing.get_context("spawn")
>>> %time ctx.Pool(1)
CPU times: user 574 µs, sys: 4.23 ms, total: 4.81 ms
Wall time: 4.34 ms
<multiprocessing.pool.Pool at 0x7fcc9450ed68> |
I ran a benchmark on the master branch on Linux (Python compiled in release mode): Result: So yes, the creation of the Pool object becomes much slower. But if you use the Pool, the overall performance shouldn't change. It's just that currently, the worker startup time it delayed to the first usage of the pool. Do you think that the slowdown is an issue? Would you prefer to wait until the worker started before terminating it, to avoid the race condition? Note: If you missed it, I tried a different approach but it changed the lifetime of handles in the parent process and so I abandonned it. |
|
Yes, I think the slowdown is definitely an issue. |
It blocks my PR #7827 which may allow to catch bugs in other Python modules, like the one that I fixed in bz2 and lzma. |
On Windows, if a worker is terminated too quickly, handles created by
DupHandle() on reduction.dump() remain open in the parent process,
causing a handles leak. Pool now waits using events until all new
workers started to make sure that the workers closed handles
inherited by DupHandle.
https://bugs.python.org/issue33966