Issue 21201: Uninformative error message in multiprocessing.Manager()

While using multiprocessing.Manager() to send data between processes I've noticed that one of the traceback messages is not very informative:

Traceback (most recent call last):
  File "age_predict.py", line 39, in <module>
    train_data, train_target, test_data = prepare_data()
  File "age_predict.py", line 30, in prepare_data
    train_data = q.get(True, 10000)
  File "<string>", line 2, in get
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 777, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------
Unserializable message: ('#RETURN', [A WHOLE LOT OF DATA GOES HERE]
 

The real problem here is that my machine is running out of memory, but the traceback message shadows this information and reports "Unserializable message" instead.

After a simple path (see: attached file) the traceback message is as follows:


Traceback (most recent call last):
  File "age_predict.py", line 39, in <module>
    train_data, train_target, test_data = prepare_data()
  File "age_predict.py", line 30, in prepare_data
    train_data = q.get(True, 10000)
  File "<string>", line 2, in get
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 775, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 288, in serve_client
    send(msg)
MemoryError: out of memory


Here's another example (see: http://bugs.python.org/issue6766):

This python3 code:

from multiprocessing import Manager
manager = Manager()
ns_proxy = manager.Namespace()
evt_proxy = manager.Event()
ns_proxy.my_event_proxy = evt_proxy
print(ns_proxy.my_event_proxy)


Produces following traceback message:


Traceback (most recent call last):
  File "test.py", line 6, in <module>
    print(ns_proxy.my_event_proxy)
  File "/cpython/Lib/multiprocessing/managers.py", line 1032, in __getattr__
    return callmethod('__getattribute__', (key,))
  File "/cpython/Lib/multiprocessing/managers.py", line 748, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
----------------------------------------------------------------------
Unserializable message: ('#RETURN', <threading.Event object at 0x7f9484aff278>)
----------------------------------------------------------------------


...and after applying the attached patch it becomes clearer what's going on:


Traceback (most recent call last):
  File "test.py", line 6, in <module>
    print(ns_proxy.my_event_proxy)
  File "/cpython/Lib/multiprocessing/managers.py", line 1032, in __getattr__
    return callmethod('__getattribute__', (key,))
  File "/cpython/Lib/multiprocessing/managers.py", line 748, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
----------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
  File "/cpython/Lib/multiprocessing/managers.py", line 276, in serve_client
    send(msg)
  File "/cpython/Lib/multiprocessing/connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "/cpython/Lib/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class '_thread.lock'>: attribute lookup lock on _thread failed

This patch doesn't break any tests.

Can someone review the patch with a view to commit please.  It's a change to one line as explained in msg215934.

Looks good to me. Perhaps we can use utils.info() to print the exception.

Looks good to me as well.  I'll go ahead with applying the patch.

Berker:  I'm not sure it's worth adding something like `util.info("Serialization failure on: %r", msg)`.  If serialization failed for some reason other than MemoryError, we will be communicating the output of format_exc() back to the client.  If serialization failed due to a MemoryError (which has tripped up plenty of people) then our util.info may similarly fail.



Note that with the patch recently applied to issue6766, under 3.6 the second example no longer triggers an exception.  That second example can even continue:
    >>> ns_proxy.my_event_proxy.set()
    >>> ns_proxy.my_event_proxy.is_set()
    True
    >>> evt_proxy.is_set()
    True
    >>> evt_proxy.clear()
    >>> ns_proxy.my_event_proxy.is_set()
    False

New changeset cbf81969aba4 by Davin Potts in branch '2.7':
Issue #21201: Improves readability of multiprocessing error message from server to client for certain exceptions
https://hg.python.org/cpython/rev/cbf81969aba4

New changeset 9b1f8c68de4c by Davin Potts in branch '3.5':
Issue #21201: Improves readability of multiprocessing error message from server to client for certain exceptions
https://hg.python.org/cpython/rev/9b1f8c68de4c

New changeset 199aca18d9a1 by Davin Potts in branch 'default':
Issue #21201: Improves readability of multiprocessing error message from server to client for certain exceptions
https://hg.python.org/cpython/rev/199aca18d9a1

Thanks for your patch @wojtekwalczak!

stage: patch review -> resolved

messages: + msg259110

stage: patch review

versions: - Python 3.1, Python 3.2, Python 3.3