Issue 30017: zlib.error: Error -2 while flushing: inconsistent stream state

I had some statements inside a `with` statement to write data to an entry in a
ZipFile. It worked great. I added a second `with` statement containing almost
exactly the same statements. That still worked great.

I refactored those common statements into a function and called that function
from the two `with` statements... and got an exception:

  zlib.error: Error -2 while flushing: inconsistent stream state

I can't figure out why it matters whether the writing happens in the `with` or
in the function called by the `with`, but here's a trimmed-down version of the
code that demonstrates the problem:

--------------------------------------------------------------------------------
#!/usr/bin/env python

import io, pprint, zipfile
from zipfile import ZIP_DEFLATED

def printLiteral( data, out ) :
        encoder = io.TextIOWrapper( out, encoding='utf-8', write_through=True )
        pprint.pprint( data, stream=encoder )

data = { 'not' : 'much', 'just' : 'some K \N{RIGHTWARDS WHITE ARROW} V pairs' }

with zipfile.ZipFile( 'zzz.zip', mode='w', compression=ZIP_DEFLATED ) as myzip :

    with myzip.open( 'this one works', 'w' ) as out :
        encoder = io.TextIOWrapper( out, encoding='utf-8', write_through=True )
        pprint.pprint( data, stream=encoder )

    with myzip.open( 'this one fails', 'w' ) as out :
        printLiteral( data, out )

        print( 'printed but entry still open' )
    print( 'entry has been closed but not file' )
print( 'zip file has been closed' )
--------------------------------------------------------------------------------

And here's the output on my Arch Linux 64bit with package `python 3.6.0-2`...
A co-worker sees the same behavior on MacOS 10.11.6 Python 3.6.1 :

--------------------------------------------------------------------------------
printed but entry still open
Traceback (most recent call last):
  File "zzz.py", line 21, in <module>
    print( 'printed but entry still open' )
  File "/usr/lib/python3.6/zipfile.py", line 995, in close
    buf = self._compressor.flush()
zlib.error: Error -2 while flushing: inconsistent stream state
--------------------------------------------------------------------------------

I tried debugging this in PyDev but got lost. Turning off the compression makes
the exception go away.

At a guess, when encoder goes out of scope, it closes the underlying file object. Then, on exiting the with block, the zipfile tries to take some action with the closed file and errors out?

It looks like the zip entry writer object may not expect its “close” method to be called multiple times. Other file objects tend to allow this with no ill effect. Adding Serhiy and Thomas who implemented the writer object in Issue 26039.

The first time it is called will be when the context manager (“with” statement) exits.

The second time will be when the TextIOWrapper is garbage-collected. This is documented behaviour <https://docs.python.org/3.6/library/io.html#io.IOBase.__del__>, but I think it is a design mistake of the IOBase hierarchy; see Issue 19829.

A reasonable workaround would be to call encoder.detach() after you have finished writing the zip entry.

I agree, that close() should be an idempotent operation. Proposed patch fixes this.

New changeset 4c0d9ea995da595e90e08813b89510de59907802 by Serhiy Storchaka in branch 'master':
bpo-30017: Allowed calling the close() method of the zip entry writer object (#1041)
https://github.com/python/cpython/commit/4c0d9ea995da595e90e08813b89510de59907802

New changeset 8e5b52a8da07e781bda50ba0a7065b1058495a37 by Serhiy Storchaka in branch '3.6':
bpo-30017: Allowed calling the close() method of the zip entry writer object (#1041) (#1092)
https://github.com/python/cpython/commit/8e5b52a8da07e781bda50ba0a7065b1058495a37

assignee: serhiy.storchaka
components: + Library (Lib)
stage: needs patch -> patch review

messages: + msg291310
stage: needs patch