Issue 26732: multiprocessing sentinel resource leak

The sentinel creates a named pipe, but the parent's end of the pipe is inherited by subsequently created children.  

import multiprocessing,signal,sys
def sproc(x): signal.pause()
for each in range(int(sys.argv[1])):
  multiprocessing.Process(target=sproc, args=(each,)).start()
signal.pause()

Running the above on Linux with varying numbers of child processes (expressed as the argument to the above) and using techniques like "$ sudo ls /proc/NNNN/fd" it is possible to see an ever growing number of pipe connections for subsequent children.

(Sorry, an unnamed pipe, but a pipe nonetheless.)

I confirm the issue with Python 3.6 on Linux.

File descriptors of the parent process:

haypo@selma$ ls -l /proc/31564/fd/
lrwx------. 1 haypo haypo 64 13 avril 00:55 0 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:55 1 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:55 10 -> pipe:[697354]
lr-x------. 1 haypo haypo 64 13 avril 00:55 11 -> pipe:[697355]
lr-x------. 1 haypo haypo 64 13 avril 00:55 12 -> pipe:[697356]
lrwx------. 1 haypo haypo 64 13 avril 00:54 2 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:55 3 -> pipe:[697347]
lr-x------. 1 haypo haypo 64 13 avril 00:55 4 -> pipe:[697348]
lr-x------. 1 haypo haypo 64 13 avril 00:55 5 -> pipe:[697349]
lr-x------. 1 haypo haypo 64 13 avril 00:55 6 -> pipe:[697350]
lr-x------. 1 haypo haypo 64 13 avril 00:55 7 -> pipe:[697351]
lr-x------. 1 haypo haypo 64 13 avril 00:55 8 -> pipe:[697352]
lr-x------. 1 haypo haypo 64 13 avril 00:55 9 -> pipe:[697353]

File descriptors of the first child process:

haypo@selma$ ls -l /proc/31565/fd/
lrwx------. 1 haypo haypo 64 13 avril 00:54 0 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:54 1 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:54 2 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:54 3 -> /dev/null
l-wx------. 1 haypo haypo 64 13 avril 00:54 4 -> pipe:[697347]

File descriptors of the second child process:

haypo@selma$ ls -l /proc/31566/fd/
lrwx------. 1 haypo haypo 64 13 avril 00:54 0 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:54 1 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:54 2 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:54 3 -> pipe:[697347]
lr-x------. 1 haypo haypo 64 13 avril 00:54 4 -> /dev/null
l-wx------. 1 haypo haypo 64 13 avril 00:54 5 -> pipe:[697348]

(...)

File descriptors of the last child process:

haypo@selma$ ls -l /proc/31574/fd/
lrwx------. 1 haypo haypo 64 13 avril 00:57 0 -> /dev/pts/0
lrwx------. 1 haypo haypo 64 13 avril 00:57 1 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:57 10 -> pipe:[697354]
lr-x------. 1 haypo haypo 64 13 avril 00:57 11 -> pipe:[697355]
lr-x------. 1 haypo haypo 64 13 avril 00:57 12 -> /dev/null
l-wx------. 1 haypo haypo 64 13 avril 00:57 13 -> pipe:[697356]
lrwx------. 1 haypo haypo 64 13 avril 00:54 2 -> /dev/pts/0
lr-x------. 1 haypo haypo 64 13 avril 00:57 3 -> pipe:[697347]
lr-x------. 1 haypo haypo 64 13 avril 00:57 4 -> pipe:[697348]
lr-x------. 1 haypo haypo 64 13 avril 00:57 5 -> pipe:[697349]
lr-x------. 1 haypo haypo 64 13 avril 00:57 6 -> pipe:[697350]
lr-x------. 1 haypo haypo 64 13 avril 00:57 7 -> pipe:[697351]
lr-x------. 1 haypo haypo 64 13 avril 00:57 8 -> pipe:[697352]
lr-x------. 1 haypo haypo 64 13 avril 00:57 9 -> pipe:[697353]

This is expected with the "fork" method, but also happens with the "forkserver" method.

New changeset 896145d9d266ee2758cfcd7691238cbc1f9e1ab8 by Antoine Pitrou in branch 'master':
bpo-26732: fix too many fds in processes started with the "forkserver" method (#2813)
https://github.com/python/cpython/commit/896145d9d266ee2758cfcd7691238cbc1f9e1ab8

The newly added test now hangs on almost all Windows buildbots, see:
http://bugs.python.org/issue31009#msg299050

By the way, 3.6 is marked as affected, but the change wasn't backported to 3.6?

Thank you for the fix, pitrou!

Regarding your comment regarding the behavior being expected with the "fork" method, I would suggest a documentation modification in the description of the fork method to add something to the effect of "All resources of the parent are inherited by the child process; this includes the ability of later children to act as the parent of earlier children via the various functions in this package."

Antoine: Do you want to backport your change 896145d9d266ee2758cfcd7691238cbc1f9e1ab8 to Python 3.6? If yes, don't forget to include my fix 302bbbe9ba5c72559913e2ea006c921f698a729d for Windows (bpo-31009).

If you prefer, I can handle the backport.

Victor, you can try to backport the fix, it should be interesting, though I suspect you will run into conflicts :-)

Kevin:

> Regarding your comment regarding the behavior being expected with the "fork" method, I would suggest a documentation modification in the description of the fork method to add something to the effect of "All resources of the parent are inherited by the child process; this includes the ability of later children to act as the parent of earlier children via the various functions in this package."

I would rather keep the latter part ("the ability of later children to act as the parent of earlier children via the various functions in this package") an implementation detail rather than a documented feature.  Who knows, perhaps some day we'll change the implementation so that those are not inherited anymore.

IIRC sockets can be marked as inheritable or not. It seems to me it would be a useful enhancement to allow the same determination for the pipes.

Hi Antoine,

> ... an implementation detail rather than a documented feature

I understand your desire, but this is a leaky abstraction and I would still suggest that it should be more clear that the fork method will inherit *all* resources from the parent, including multiprocessing internal resources related to other processes.  This is potentially important if there are (a) large numbers of processes to be created and (b) for security aspects.

Regards,
  Kevin

messages: + msg298844

stage: needs patch