Issue 28131: assert statements missed when loaded by zipimporter

Grabbing the recently released Python 3.6.0b1, I tried running one of my test suites, but found that some assertions were failing to assert when the package was loaded as a zip file (such as with pytest-runner installed dependencies). I distilled the issue to this:

$ cat > mod.py
def test(val):
        assert(val)
        print(val)
$ zip mod.zip mod.py
updating: mod.py (deflated 20%)
$ rm mod.py
$ python
Python 3.6.0b1 (v3.6.0b1:5b0ca4ed5e2f, Sep 12 2016, 09:24:46) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('mod.zip')
>>> import mod
>>> mod.test(False)
False
>>> mod.__loader__
<zipimporter object "mod.zip">
>>> sys.flags.optimize
0

I would have expected the call to mod.test to have raised an AssertionError, and on Python 3.5 it does.

I searched the what's new and didn't see anything advertising this change, so I suspect it's an unintentional regression.

I'm including Brett for his familiarity with importlib, but welcome re-assignment.

If I can do more to help, let me know.

I've added Greg and Thomas in case they have any ideas as they have looked at zipimport more recently than I have.

This shouldn't be happening and makes no sense.  It looks like the assert statement was removed at import code compilation time given the pdb trace with it from a zip file vs with it outside of a zip file:


>>> pdb.run('mod.test(False)')
> <string>(1)<module>()
(Pdb) n
False
--Return--
> <string>(1)<module>()->None
(Pdb) n
>>> pdb.run('mod.test(False)')
> <string>(1)<module>()->None
(Pdb) s
--Call--
> oss/cpython/build/mod.zip/mod.py(1)test()
-> def test(val):
(Pdb) s
> oss/cpython/build/mod.zip/mod.py(3)test()
-> print(val)
(Pdb) s
False
--Return--
> oss/cpython/build/mod.zip/mod.py(3)test()->None
-> print(val)
(Pdb) s
--Return--
> <string>(1)<module>()->None
(Pdb) s
> oss/cpython/default/Lib/bdb.py(435)run()
-> self.quitting = True
(Pdb) s
>>> 

vs no zip file:

>>> pdb.run('mod.test(False)')
> <string>(1)<module>()
(Pdb) s
--Call--
> oss/cpython/build/mod.py(1)test()
-> def test(val):
(Pdb) s
> oss/cpython/build/mod.py(2)test()
-> assert(val)
(Pdb) s
AssertionError
> oss/cpython/build/mod.py(2)test()
-> assert(val)
(Pdb) s
--Return--
> oss/cpython/build/mod.py(2)test()->None
-> assert(val)
(Pdb) s
AssertionError
> <string>(1)<module>()
(Pdb) s
--Return--
> <string>(1)<module>()->None
(Pdb) s
AssertionError
> oss/cpython/default/Lib/bdb.py(431)run()
-> exec(cmd, globals, locals)
(Pdb) s
> oss/cpython/default/Lib/bdb.py(432)run()
-> except BdbQuit:
(Pdb) s
> oss/cpython/default/Lib/bdb.py(435)run()
-> self.quitting = True
(Pdb) s
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "oss/cpython/default/Lib/pdb.py", line 1568, in run
    Pdb().run(statement, globals, locals)
  File "oss/cpython/default/Lib/bdb.py", line 431, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "oss/cpython/build/mod.py", line 2, in test
    assert(val)
AssertionError

from the zip:

>>> dis.dis(mod.test)
  3           0 LOAD_GLOBAL              0 (print)
              2 LOAD_FAST                0 (val)
              4 CALL_FUNCTION            1
              6 POP_TOP
              8 LOAD_CONST               0 (None)
             10 RETURN_VALUE

from the filesystem:
>>> dis.dis(mod.test)
  2           0 LOAD_FAST                0 (val)
              2 POP_JUMP_IF_TRUE         8
              4 LOAD_GLOBAL              0 (AssertionError)
              6 RAISE_VARARGS            1

  3     >>    8 LOAD_GLOBAL              1 (print)
             10 LOAD_FAST                0 (val)
             12 CALL_FUNCTION            1
             14 POP_TOP
             16 LOAD_CONST               0 (None)
             18 RETURN_VALUE

This is introduced in 663a62bcf9c9. The compile optimize flag is opened in the change.

Bah, that's meant to be a -1 in that change, not 1. I must have typo'd when moving from the default branch to put it in 3.5.

Anyone's welcome to fix it, or I'll get it when I have time for CPython work.

Here is a patch with a test case.

lgtm

New changeset 7bec326972f5 by Berker Peksag in branch '3.5':
Issue #28131: Fix a regression in zipimport's compile_source()
https://hg.python.org/cpython/rev/7bec326972f5

New changeset 7a6c0c4e6072 by Berker Peksag in branch '3.6':
Issue #28131: Merge from 3.5
https://hg.python.org/cpython/rev/7a6c0c4e6072

New changeset 873760b02024 by Berker Peksag in branch 'default':
Issue #28131: Merge from 3.6
https://hg.python.org/cpython/rev/873760b02024

Thanks for the report and for the reviews!

stage: patch review -> resolved

type: behavior

keywords: + patch
nosy: + berker.peksag
messages: + msg276372
stage: patch review

versions: + Python 3.5