Issue 38691: importlib: PYTHONCASEOK should be ignored when using python3 -E

When using python3 -E or python3 -I, PYTHONCASEOK environment variable should be ignored by importlib. See an email sent in 2012:
https://mail.python.org/pipermail/python-dev/2012-December/123403.html

See importlib._bootstrap_external._relax_case attribute and its _make_relax_case() function.

sys.flags.ignore_environment should be used.

Created this PR: GH-18314

@Ido Michael
I left review comments for you :)
Thanks for the contribution.

New changeset d83b6600b25487e4ebffd7949d0f478de9538875 by idomic in branch 'master':
bpo-38691 Added a switch to ignore PYTHONCASEOK when -E or -I flags passed (#18314)
https://github.com/python/cpython/commit/d83b6600b25487e4ebffd7949d0f478de9538875

Tests fail on macOS:
https://buildbot.python.org/all/#/builders/275/builds/249

I reopen the issue. The issue should be fixed soon, or the change will be reverted to repair buildobts:
https://pythondev.readthedocs.io/ci.html#revert-on-fail

======================================================================
FAIL: test_case_insensitivity (test.test_importlib.extension.test_case_sensitivity.Frozen_ExtensionModuleCaseSensitivityTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/extension/test_case_sensitivity.py", line 36, in test_case_insensitivity
    self.assertTrue(hasattr(loader, 'load_module'))
AssertionError: False is not true

======================================================================
FAIL: test_case_insensitivity (test.test_importlib.extension.test_case_sensitivity.Source_ExtensionModuleCaseSensitivityTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/extension/test_case_sensitivity.py", line 36, in test_case_insensitivity
    self.assertTrue(hasattr(loader, 'load_module'))
AssertionError: False is not true

======================================================================
FAIL: test_insensitive (test.test_importlib.source.test_case_sensitivity.Frozen_CaseSensitivityTestPEP302)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/source/test_case_sensitivity.py", line 57, in test_insensitive
    self.assertIsNotNone(insensitive)
AssertionError: unexpectedly None

======================================================================
FAIL: test_insensitive (test.test_importlib.source.test_case_sensitivity.Frozen_CaseSensitivityTestPEP451)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/source/test_case_sensitivity.py", line 57, in test_insensitive
    self.assertIsNotNone(insensitive)
AssertionError: unexpectedly None

======================================================================
FAIL: test_insensitive (test.test_importlib.source.test_case_sensitivity.Source_CaseSensitivityTestPEP302)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/source/test_case_sensitivity.py", line 57, in test_insensitive
    self.assertIsNotNone(insensitive)
AssertionError: unexpectedly None

======================================================================
FAIL: test_insensitive (test.test_importlib.source.test_case_sensitivity.Source_CaseSensitivityTestPEP451)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/source/test_case_sensitivity.py", line 57, in test_insensitive
    self.assertIsNotNone(insensitive)
AssertionError: unexpectedly None

I'm unable to debug the issue on macOS. I prepared PR 18553 to revert the change, just to give more time to fix the issue. It's to repair the CI, so we can notice other regressions.

> Tests fail on macOS: https://buildbot.python.org/all/#/builders/275/builds/249

Tests are run with "./python.exe  ./Tools/scripts/run_tests.py -j 1 -u all -W --slowest --fail-env-changed --timeout=900 -j2 --junit-xml test-results.xml" which runs tests with "./python.exe -E": ignore PYTHON* environment variables. Maybe the test should just take that in account.

Yes I saw those in the morning, thanks for patching it up. I will debug this over the weekend and will update.

From what I can tell, the regression seems like it could be fixed by adding "@unittest.skipIf(sys.flags.ignore_environment)" to the following tests in python/cpython/Lib/test/test_importlib/source/test_case_sensitivity.py that modify "PYTHONCASEOK":

CaseSensitivityTest.test_sensitive
CaseSensitivityTest.test_insensitive

Those tests seem ultimately pointless if environmental variables are being ignored (-E or -I), so I think in this case it's a misleading test failure, no?

New changeset 4dee92b0ad9f4e3ea2fbbbb5253340801bb92dc7 by Victor Stinner in branch 'master':
Revert "bpo-38691 Added a switch to ignore PYTHONCASEOK when -E or -I flags passed (#18314)" (GH-18553)
https://github.com/python/cpython/commit/4dee92b0ad9f4e3ea2fbbbb5253340801bb92dc7

> Yes I saw those in the morning, thanks for patching it up. I will debug this over the weekend and will update.

Ok, perfect. In the meanwhile, as announced, I reverted the change to be able to notify other regressions.

Once we get a fix, we can reapply the change with the fix.

Added a new clean PR with the code changes, will let you know once the tests are fixed: GH-18627

@vstinner ready for review.

New changeset fc72ab6913f2b5337ae7fda711f2de846d38f479 by idomic in branch 'master':
bpo-38691: importlib ignores PYTHONCASEOK if -E is used (GH-18627)
https://github.com/python/cpython/commit/fc72ab6913f2b5337ae7fda711f2de846d38f479

Re-opening as it causes refleaks all across the stable buildbots. I can reproduce on macOS Catalina as well. Run this to see for yourself:

- ./python.exe -E -Wd -m test -uall,-gui -l -L -R: test_importlib

Reverting GH-18627 fixes the issue.

> Reverting GH-18627 fixes the issue.

Sorry, this is false. I just checked out to the last commit to importlib before GH-18627 and it did not refleak. But when I actually reverted just GH-18627, the issue persisted.

By bisecting, I found out that it's GH-19084 which causes the refleak and it only just so happens that this refleak appears in importlib tests.

FYI importlib leak was fixed in bpo-40050 by:

commit 83d46e0622d2efdf5f3bf8bf8904d0dcb55fc322
Author: Victor Stinner <vstinner@python.org>
Date:   Tue Mar 24 18:03:34 2020 +0100

    bpo-40050: Fix importlib._bootstrap_external (GH-19135)
    
    Remove two unused imports: _thread and _weakref. Avoid creating a new
    winreg builtin module if it's already available in sys.modules.
    
    The winreg module is now stored as "winreg" rather than "_winreg".

stage: commit review -> resolved

nosy: + lukasz.langa
messages: + msg364843

keywords: + needs review, - patch
resolution: fixed -> (no value)

messages: + msg362226
title: [easy] importlib: PYTHONCASEOK should be ignored when using python3 -E -> importlib: PYTHONCASEOK should be ignored when using python3 -E

messages: + msg360608
title: importlib: PYTHONCASEOK should be ignored when using python3 -E -> [easy] importlib: PYTHONCASEOK should be ignored when using python3 -E