Issue 33169: importlib.invalidate_caches() doesn't clear all caches

See https://github.com/python/mypy/pull/4811. To summarize, importlib.invalidate_caches() doesn't clear the negative cache in sys.path_importer_cache.

Could be related to https://bugs.python.org/issue30891?

Correct because as documented it calls invalidate_caches() on meta-path finders and the one that handles sys.path calls invalidate_caches() on the finders found in sys.path_importer_cache() (as documented by https://docs.python.org/3/library/importlib.html#importlib.machinery.PathFinder.invalidate_caches). Since None isn't a finder it doesn't do anything there as there is no method to call. So the "full of lies" comment by the OP on the mypy repo is over-the-top/incorrect.

Now tweaking https://github.com/python/cpython/blob/521995205a2cb6b504fe0e39af22a81f785350a3/Lib/importlib/_bootstrap_external.py#L1180-L1186 to delete entries in sys.path_importer_cache() that are set to None is straight-forward, but I don't think we can backport since it would be going against what the documentation says it does.

I agree that we can't change 3.6 at this late stage, but the change is borderline enough on the "new feature"/"bug fix" scale that I think it's worth asking Ned if he's comfortable with us changing it for 3.7.0b4.

Ned: the change we'd like to make is to update PathFinder.invalidate_caches to delete all entries in sys.path_importer_cache that have a value of None.

The current behavior is causing problems for at least some use cases so it does seems to be closer to a bug than a feature.  If there is agreement that it is desirable to make the change, I think it's better to do it now in 3.7 than later.

SGTM.

On Fri, Mar 30, 2018, 06:41 Ned Deily <report@bugs.python.org> wrote:

>
> Ned Deily <nad@python.org> added the comment:
>
> The current behavior is causing problems for at least some use cases so it
> does seems to be closer to a bug than a feature.  If there is agreement
> that it is desirable to make the change, I think it's better to do it now
> in 3.7 than later.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue33169>
> _______________________________________
>

OK, I will update importlib.machinery.PathFinder.invalidate_caches() to also drop None from sys.path_importer_cache (objects that are not None and do not define invalidate_caches() will be left alone just like they are today). I should hopefully be able to get to this before b4.

New changeset 9e2be60634914f23db2ae5624e4acc9335bf5fea by Brett Cannon in branch 'master':
bpo-33169: Remove values of `None` from sys.path_importer_cache when invalidating caches (GH-6402)
https://github.com/python/cpython/commit/9e2be60634914f23db2ae5624e4acc9335bf5fea

New changeset a09bb87c1eebb07b01b8105cf536704893aec565 by Brett Cannon in branch '3.7':
[3.7] bpo-33169: Remove values of `None` from sys.path_importer_cache when invalidating caches (GH-6402) (GH-6403)
https://github.com/python/cpython/commit/a09bb87c1eebb07b01b8105cf536704893aec565

All done in 3.7 and master.

stage: patch review -> resolved