Issue 39791: New `files()` api from importlib_resources.

In the [importlib_resources backport](https://gitlab.com/python-devs/importlib_resources/)... in particular in [issue 58](https://gitlab.com/python-devs/importlib_resources/issues/58) and [merge request 76](https://gitlab.com/python-devs/importlib_resources/-/merge_requests/76), the backport now has a new feature, a "files()" function.

Let's incorporate that functionality into importlib.resources.

The latest release, 1.3.0, includes extensibility support and has been merged with the cpython branch of the importlib_resources project. I believe that code is now synced with this project and ready to be applied here. I'm hoping benthayer can apply the changes and submit the pr.

New changeset 7f7e706d78ab968a1221c6179dfdba714860bd12 by Jason R. Coombs in branch 'master':
bpo-39791: Add files() to importlib.resources (GH-19722)
https://github.com/python/cpython/commit/7f7e706d78ab968a1221c6179dfdba714860bd12

I've merged PR 19722. Some follow up actions I'd like to do:

- Add hooks for `.files()` on built-in loaders.
- Replace `loader.get_resource_reader()` with adapters around `.files()` for built-in loaders.

New changeset 161541ab45278df6603dd870113b10f13e4d9e16 by Jason R. Coombs in branch 'master':
bpo-39791: Refresh importlib.metadata from importlib_metadata 1.6.1. (GH-20659)
https://github.com/python/cpython/commit/161541ab45278df6603dd870113b10f13e4d9e16

New changeset a4fa9a95153a3800dea60b3029b2dcaf8a4f6acb by Miss Islington (bot) in branch '3.9':
bpo-39791: Refresh importlib.metadata from importlib_metadata 1.6.1. (GH-20659) (GH-20661)
https://github.com/python/cpython/commit/a4fa9a95153a3800dea60b3029b2dcaf8a4f6acb

The 'fancy' file name breaks on latin-1 character set.

aixtools@gcc119:[/home/aixtools/python/py39-3.9] a4fa9a95153a3800dea60b3029b2dcaf8a4f6acb Lib/test/test_importlib/test_main.py                   <
diff --git a/Lib/test/test_importlib/test_main.py b/Lib/test/test_importlib/test_main.py
index 42a7999..7b18c3d 100644
--- a/Lib/test/test_importlib/test_main.py
+++ b/Lib/test/test_importlib/test_main.py
@@ -246,3 +246,19 @@ class TestEntryPoints(unittest.TestCase):
         """
         with self.assertRaises(Exception):
             json.dumps(self.ep)
+
+    def test_module(self):
+        assert self.ep.module == 'value'
+
+    def test_attr(self):
+        assert self.ep.attr is None
+
+
+class FileSystem(fixtures.OnSysPath, fixtures.SiteDir, unittest.TestCase):
+    def test_unicode_dir_on_sys_path(self):
+        """
+        Ensure a Unicode subdirectory of a directory on sys.path
+        does not crash.
+        """
+        fixtures.build_files({'☃': {}}, prefix=self.site_dir)
+        list(distributions())

AIX bots fail with:

Captured traceback
==================

Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.9.edelsohn-aix-ppc64/build/Lib/test/test_importlib/test_main.py", line 263, in test_unicode_dir_on_sys_path
    fixtures.build_files({'\u2603': {}}, prefix=self.site_dir)
  File "/home/shager/cpython-buildarea/3.9.edelsohn-aix-ppc64/build/Lib/test/test_importlib/fixtures.py", line 202, in build_files
    full_name.mkdir()
  File "/home/shager/cpython-buildarea/3.9.edelsohn-aix-ppc64/build/Lib/pathlib.py", line 1309, in mkdir
    self._accessor.mkdir(self, mode)
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2603' in position 17: ordinal not in range(256)



Test report
===========

Failed tests:

- test_importlib

Thanks for the report Michael. I'm trying to figure out the best way to address the issue. That test is shared with importlib_metadata, so needs to run without CPython's test suite fixtures, such as the ones that generate non-ascii filenames. I'm tempted to just attempt to create the fixture and skip if the fixture fails. An alternate approach might be to attempt to load CPython's fixture, skip if that value is None, and fallback to the snowman otherwise.

See GH-20681 for a proposed fix. I've scheduled the build bots to run the patch. Will build bots prove the fix? If not, can you test the patch in the same environment where it was discovered?

New changeset 2efe18bf277dd0f38a1d248ae6bdd30947c26880 by Jason R. Coombs in branch 'master':
bpo-39791: Support file systems that cannot support non-ascii filenames (skipping tests in that case). (#20681)
https://github.com/python/cpython/commit/2efe18bf277dd0f38a1d248ae6bdd30947c26880

New changeset 71f501698d64af6463246bc6efdbbf1b85616102 by Miss Islington (bot) in branch '3.9':
bpo-39791: Support file systems that cannot support non-ascii filenames (skipping tests in that case). (GH-20681)
https://github.com/python/cpython/commit/71f501698d64af6463246bc6efdbbf1b85616102

New changeset 843c27765652e2322011fb3e5d88f4837de38c06 by Jason R. Coombs in branch 'master':
bpo-39791 native hooks for importlib.resources.files (GH-20576)
https://github.com/python/cpython/commit/843c27765652e2322011fb3e5d88f4837de38c06

New changeset 9cf1be46e3692d565461afd3afa326d124d743dd by Miss Islington (bot) in branch '3.9':
bpo-39791 native hooks for importlib.resources.files (GH-20576)
https://github.com/python/cpython/commit/9cf1be46e3692d565461afd3afa326d124d743dd

Note that the most recent commits have introduced a critical regression in importlib.resources.path() that breaks the certifi package from PyPI and presumably other users of path().  See Issue40924.

New changeset ce5e6f098f8a270e50b989baa75765584573706b by Łukasz Langa in branch '3.9':
[3.9] bpo-40924: Revert "bpo-39791 native hooks for importlib.resources.files (GH-20576)" (#20760)
https://github.com/python/cpython/commit/ce5e6f098f8a270e50b989baa75765584573706b

> bpo-39791 native hooks for importlib.resources.files (GH-20576)
> https://github.com/python/cpython/commit/843c27765652e2322011fb3e5d88f4837de38c06

This change introduced bpo-41007 "test_importlib logs ResourceWarning: test_path.CommonTests.test_importing_module_as_side_effect()" regression.

This issue was implemented, just not as fully as I'd have hoped. Still lacking is native support for .files on the built-in package providers and removing the legacy APIs or at least configuring them to rely on the files API, but as far as supplying the files API, this work is done.

pull_requests: + pull_request23393

pull_requests: + pull_request23392

resolution: fixed
stage: patch review -> resolved

messages: + msg371744
versions: + Python 3.10, - Python 3.9