Issue 27144: concurrent.futures.as_completed() memory inefficiency

as_complite generator keeps reference of all passed futures until StopIteration. It may lead to serious memory inefficiency.

Solution is to remove reference from lists and yield future ad-hoc.

I have submitted patch and reproduce sample.

I can create backport for older versions if needed.

Is there a reason this hasn't been merged yet?

Fixing this bug locally (albeit just by setting Future._result = None when I've extracted the result) reduced the memory footprint of my program from 50 GB to 7 GB, so it seems worth it.

Any update on this reviewing, patching, and then backporting to 2.7? We've just run into the exact same issue here in Google using a ThreadPoolExecutor.map call growing memory usage without bounds due to the Future holding onto its result even after being returned from the map call. There's a more specific patch possible for just map() itself, but this more general one seems like it'd be better.

> Any update on this reviewing, patching, and then backporting to 2.7?

concurrent package was added in 3.2. How backport to 2.7? Nosy myself.

> We just ran into the exact same issue here in Google using a ThreadPoolExecutor.map call
Looks like map got simillar issue. I created GitHub PR with map fixed.

>  Concurrent package was added in 3.2. How backport it 2.7?
There is official, unofficial 2.7 backport.

Git: https://github.com/agronholm/pythonfutures
Pip: futures
Ubuntu package: python-concurrent.futures

New changeset 97e1b1c81458d2109b2ffed32ffa1eb643a6c3b9 by Antoine Pitrou (Grzegorz Grzywacz) in branch 'master':
bpo-27144: concurrent.futures as_complete and map iterators do not keep reference to returned object (#1560)
https://github.com/python/cpython/commit/97e1b1c81458d2109b2ffed32ffa1eb643a6c3b9

New changeset ea767915f7476c1fe97f7b1a53304d57f105bdd2 by Antoine Pitrou in branch '3.6':
[3.6] bpo-27144: concurrent.futures as_complete and map iterators do not keep reference to returned object (GH-1560) (#3266)
https://github.com/python/cpython/commit/ea767915f7476c1fe97f7b1a53304d57f105bdd2

I've merged the PR to master and backported it to 3.6.  Thank you Grzegorz for contributing this!

"concurrent.futures.as_completed() memory inefficiency" hum, sadly the commit 97e1b1c81458d2109b2ffed32ffa1eb643a6c3b9 introduced a reference leak. Example:

$ ./python -m test -R 3:3 -test_concurrent_futures -m test.test_concurrent_futures.ProcessPoolAsCompletedTests.test_zero_timeout
(...)
test_concurrent_futures leaked [27, 27, 27] references, sum=81
test_concurrent_futures leaked [16, 17, 16] memory blocks, sum=49
(...)

Thank you Victor.  There is indeed a logic error in the new code.  I'm opening a new PR.

Tests are reusing finished futures. `_yield_and_decref` function do not clear waiters in finished futures.

In the initial merge i propose to clear waiters but after review we decide it should be removed.

I am confused now, should we change tests or restore initial `_yield_and_decref` function.

New changeset 2ef37607b7aacb7c750d008b9113fe11f96163c0 by Antoine Pitrou in branch 'master':
Fix a c.f.as_completed() refleak previously introduced in bpo-27144 (#3270)
https://github.com/python/cpython/commit/2ef37607b7aacb7c750d008b9113fe11f96163c0

New changeset 5cbca0235b8da07c9454bcaa94f12d59c2df0ad2 by Antoine Pitrou in branch '3.6':
[3.6] Fix a c.f.as_completed() refleak previously introduced in bpo-27144 (GH-3270) (#3271)
https://github.com/python/cpython/commit/5cbca0235b8da07c9454bcaa94f12d59c2df0ad2

The regression should be fixed now.

Thank you both for this nice enhancement.

nosy: + vstinner
messages: + msg301181

resolution: fixed -> (no value)

resolution: fixed
stage: patch review -> resolved

messages: + msg293439
versions: + Python 3.7