Issue 25718: itertools.accumulate __reduce__/__setstate__ bug

itertools.accumulate() has got methods __reduce__() and __setstate__() which fail at saving/restoring the state in all cases.  Example:

   >>> a = itertools.accumulate([0.0, 42])
   >>> next(a)
   0.0
   >>> b = copy.deepcopy(a)
   >>> next(a)
   42.0
   >>> next(b)
   42           # should have been the same as the previous result

More precisely, the problem occurs when the C-level "total" field has some value whose truth-value is false.  Then __reduce__/__setstate__ will not restore the value of this field, but keep it NULL.  That's why in the example above the intermediate total of 0.0 is forgotten, and the next(b) call will again just pick the next value without doing any addition.  (The problem can hurt more than in this example if we use a custom function instead of addition.)

A fix is easy but, as far as I can tell, it requires breaking the pickled format of accumulate() iterators that have already started.

__reduce__() and __setstate__() methods of itertools.accumulate() look correct. The problem is not in itertools.accumulate(), but in the copy module. It uses the same __reduce__ protocol as pickle, but in different way. In the pickle module (in both Python and C implementations) state value is ignored only if it is None. In the copy module any state with boolean value is False is ignored.

Proposed patch fixes the copy module.

Ok, then with pickle you can have the same problem but only with None-vs-no-total.  Here is an artificial example:

import itertools, pickle

def foo(a, b):
    print(a, b)

a = itertools.accumulate([3, 4, 5], foo)
next(a)
next(a)   # prints: 3, 4

b = pickle.loads(pickle.dumps(a))

next(a)   # prints: None, 5
next(b)   # foo() is not called at all

In this case yes, you are unlucky. But this is an artificial example, and I don't think this happens in any real code.

However specific meaning of None should be documented (there are other wrong words about __getstate__).

New changeset 095d21df9374 by Serhiy Storchaka in branch '3.4':
Issue #25718: Fixed copying object with state with boolean value is false.
https://hg.python.org/cpython/rev/095d21df9374

New changeset 3d39e1e2dcb3 by Serhiy Storchaka in branch '2.7':
Issue #25718: Fixed copying object with state with boolean value is false.
https://hg.python.org/cpython/rev/3d39e1e2dcb3

New changeset fecb07050aae by Serhiy Storchaka in branch '3.5':
Issue #25718: Fixed copying object with state with boolean value is false.
https://hg.python.org/cpython/rev/fecb07050aae

New changeset 962086677306 by Serhiy Storchaka in branch 'default':
Issue #25718: Fixed copying object with state with boolean value is false.
https://hg.python.org/cpython/rev/962086677306

The bug in the copy module is fixed. As for accumulate's __reduce__, I don't think we can do something.

This could be fixed by saving the accumulate state in a tuple.  It would break protocol, though.  I don't recall the rules for backwards compatibility of pickles.
I've argued before that the state of runtime structures such as generators is so intimately tied with the currently executing python that we shouldn't worry to much about it.  This sort of stuff is used for caches, IPC, and so on.  But it's not my call.

Well, we can fix this without breaking the protocol. The accumulate() object with state is None can be reconstructed as islice(accumulate(chain([None], iterator)), 1, None).

Proposed patch implements this solution.

What would you say about the patch Kristján?

Ping.

Ping again.

Raymond, could you please look at the patch?

+1

New changeset f3c54cbac3de by Serhiy Storchaka in branch '3.5':
Issue #25718: Fixed pickling and copying the accumulate() iterator with total is None.
https://hg.python.org/cpython/rev/f3c54cbac3de

New changeset 49c035b30e18 by Serhiy Storchaka in branch 'default':
Issue #25718: Fixed pickling and copying the accumulate() iterator with total is None.
https://hg.python.org/cpython/rev/49c035b30e18

messages: + msg261237
nosy: + rhettinger

messages: + msg256060

messages: + msg255280

components: + Library (Lib), - Extension Modules
stage: patch review

nosy: + kristjan.jonsson