Issue 30167: site.main() does not work on Python 3.6 and superior if PYTHONSTARTUP is set

Apparently, "import site; site.main()" does not seem to work anymore on Python 3.6 and superior.

The reason is a change on the behavior of "os.path.abspath(None)". Before Python 3.6, it used to report an AttributeError which is properly caught inside "site.abs_paths" (see: https://github.com/python/cpython/blob/master/Lib/site.py#L99), making it ignore __main__, one of sys.modules, which has __file__ and __cached__ set to None.

With Python 3.6 and superior, os.path.abspath(None) reports a TypeError, which makes calling "site.main()" raise an exception and stop.

How to reproduce: On python 3.6 or superior, do "import site; site.main()".

Expected behavior: Exception is properly caught and treated inside "site.abs_paths", ignoring modules in which __file__ and/or __cached__ are set to None.

More information: to reproduce the problem, don't use Python's "-c" command-line option. In this case "__main__" won't be inside sys.modules which mitigates the issue.

After further investigation, the issue can only be reproduced iff the user sets PYTHONSTARTUP which triggers "__main__" to appear in sys.modules and the problem to occur.

ref to other projects that hit this issue.

https://gitlab.idiap.ch/bob/bob.buildout/issues/15
https://github.com/saltstack/salt/issues/45080
https://groups.google.com/forum/#!msg/comp.lang.python/tpfiHAJhl9Y/hZj1f50vBQAJ

https://github.com/python/cpython/blob/f822549653d8d09bffff5b7dcddfdf12679a787c/Python/pythonrun.c#L391-L399

__file__ and __cached__ are added here.  And,

https://github.com/python/cpython/blob/f822549653d8d09bffff5b7dcddfdf12679a787c/Python/pythonrun.c#L441-L442

Only __file__ is removed here.  I feel __cached__ should be removed too.


On the other hand, if None is valid value for __cached__, site.abs_paths() should ignore it.

https://www.python.org/dev/peps/pep-3147/#file

> It is recommended that when nothing sensible can be calculated, implementations should set the __cached__ attribute to None.

It seems it's valid, and recommended for some cases.  So PR-6731 looks OK.

New changeset 2487f30d5529948ace26559e274d7cac6abcd1a8 by Ned Deily (Steve Weber) in branch 'master':
bpo-30167: Prevent site.main() exception if PYTHONSTARTUP is set. (GH-6731)
https://github.com/python/cpython/commit/2487f30d5529948ace26559e274d7cac6abcd1a8

New changeset ec4343c3b4c0e0a7500122fac616e6488c0ab842 by Ned Deily (Miss Islington (bot)) in branch '3.7':
bpo-30167: Prevent site.main() exception if PYTHONSTARTUP is set. (GH-6731) (GH-7606)
https://github.com/python/cpython/commit/ec4343c3b4c0e0a7500122fac616e6488c0ab842

New changeset 3e121581d008a780b8a9f1bcda5966cf0c06f6d5 by Ned Deily (Miss Islington (bot)) in branch '3.6':
bpo-30167: Prevent site.main() exception if PYTHONSTARTUP is set. (GH-6731) (GH-7607)
https://github.com/python/cpython/commit/3e121581d008a780b8a9f1bcda5966cf0c06f6d5

As I noted in the discussion on PR 6731, I think there should be a test for this so we don't break it again.  But, since it seems that the problem has affected a number of users and projects and since the fix is easy and easily testable manually, I decided to merge the PR for 3.7.0rc1 and 3.6.6rc1.  I am leaving the issue open for someone to supply a PR with a test case and for discussion of PR 7415.  Thank you all!

New changeset d4c76d960b8b286b75c933780416ace9cda682fd by INADA Naoki in branch 'master':
bpo-30167: Add test for module.__cached__ is None (GH-7617)
https://github.com/python/cpython/commit/d4c76d960b8b286b75c933780416ace9cda682fd

New changeset 82daa60defbd6497efdaa6c1132ecc8563122ed5 by INADA Naoki in branch 'master':
bpo-30167: Remove __cached__ from __main__ when removing __file__ (GH-7415)
https://github.com/python/cpython/commit/82daa60defbd6497efdaa6c1132ecc8563122ed5