Issue 37951: Disallow fork in a subinterpreter broke subprocesses in mod_wsgi daemon mode

BPO https://bugs.python.org/issue34651 disabled fork in subinterpreters. The patch also disabled fork() in _posixsubprocess.fork_exec(). This broke the ability to spawn subprocesses in mod_wsgi daemons, which use subinterpreters. Any attempt to spawn (fork + exec) a subprocess fails with "RuntimeError: fork not supported for subinterpreters":

...
  File "/usr/lib64/python3.8/subprocess.py", line 829, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib64/python3.8/subprocess.py", line 1608, in _execute_child
    self.pid = _posixsubprocess.fork_exec(
RuntimeError: fork not supported for subinterpreters

Also see https://bugzilla.redhat.com/show_bug.cgi?id=1745450

subprocess still work in subinterpreters in Python 3.8 if posix_spawn() can be used, but posix_spawn() is only used under some conditions:
https://docs.python.org/dev/whatsnew/3.8.html#optimizations

"The subprocess module can now use the os.posix_spawn() function in some cases for better performance. Currently, it is only used on macOS and Linux (using glibc 2.24 or newer) if all these conditions are met:

* close_fds is false;
* preexec_fn, pass_fds, cwd and start_new_session parameters are not set;
* the executable path contains a directory."

--

It seems like FreeIPA uses ctypes and ctypes calls subprocess.Popen(['/sbin/ldconfig', '-p'], ...) to locale libcrypto.

I see different options:

* modify FreeIPA / ctypes to ensure that posix_spawn() can be used
* avoid subinterpreters to deploy FreeIPA
* revert the change to allow again fork in subprocesses: see bpo-34651 for the rationale why it was denied

I understand that FreeIPA is run as WSGI using mod_wsgi in Apache.

It's a bit more complicated. FreeIPA uses cryptography, which uses asn1crypto, which uses ctypes, which is broken in mod_wsgi due to bpo-34651. It's not just FreeIPA that is affected by the issue. Any application running in mod_wsgi is potentially affected and broken by bpo-34651.

1a) (modify FreeIPA) is not possible. IPA requires the additional features of the subprocess module.
1b) (modify ctypes) should be done in a separate ticket. I'm not sure why subprocess does not use posix_spawn() here. I guess it's the default value "close_fds=True"?
2) (avoid subinterpreters) would require a rewrite of mod_wsgi
3) (revert bpo-34651) is IMHO required for _posixsubprocess.fork_exec().

bpo-34651 is a backwards incompatible change that breaks existing applications that uses mod_wsgi. At least _posixsubprocess.fork_exec() should be reverted and the removal of fork() support should go through a proper deprecation cycle of two releases.

I'm bumping this up to release blocker and CC Łukasz.

Christian, you're right to treat this as Release Blocker. Let's have this fixed. Assigning Eric?

FWIW, _posixsubprocess.fork_exec() should be safe to allow.

The only thing within it to disallow, if you're going to bother to check this at all, is any use of the legacy preexec_fn support.

I have created a PR that implements Greg's proposal https://bugs.python.org/issue34651#msg325302

New changeset 98d90f745d35d5d07bffcb46788b50e05eea56c6 by Christian Heimes in branch 'master':
bpo-37951: Lift subprocess's fork() restriction (GH-15544)
https://github.com/python/cpython/commit/98d90f745d35d5d07bffcb46788b50e05eea56c6

New changeset 03c52f2f63a8abeb4afb75e9da46c7d6c0a8afd5 by Miss Islington (bot) in branch '3.8':
bpo-37951: Lift subprocess's fork() restriction (GH-15544)
https://github.com/python/cpython/commit/03c52f2f63a8abeb4afb75e9da46c7d6c0a8afd5

Thanks Victor and Gregory!

I'm reducing the severity from release blocker to high and keep the ticket in pending to give Eric a change to review the commits.

Well, now our (Fedora QA's) automated testing of FreeIPA is showing what looks like a problem with preexec_fn (rather than fork) being disallowed:

https://bugzilla.redhat.com/show_bug.cgi?id=1759290

Login to the FreeIPA webUI is failing, and at the time it fails we see this error message on the server end:

[Mon Oct 07 09:22:19.521604 2019] [wsgi:error] [pid 32989:tid 139746234119936] [remote 10.0.2.102:56054] ipa: DEBUG: args=['/usr/bin/kinit', 'admin', '-c', '/run/ipa/ccaches/kinit_32989', '-E']
[Mon Oct 07 09:22:19.521996 2019] [wsgi:error] [pid 32989:tid 139746234119936] [remote 10.0.2.102:56054] ipa: DEBUG: Process execution failed
[Mon Oct 07 09:22:19.522189 2019] [wsgi:error] [pid 32989:tid 139746234119936] [remote 10.0.2.102:56054] ipa: INFO: 401 Unauthorized: preexec_fn not supported within subinterpreters

preexec_fn is fundamentally unsupportable.

what code is using it, there should be a way not to rely on that.

It's this function:

https://github.com/freeipa/freeipa/blob/master/ipalib/install/kinit.py#L66

The function `run` is imported from `ipapython.ipautil`, it's defined here:

https://github.com/freeipa/freeipa/blob/master/ipapython/ipautil.py#L391

all of this is being run inside a WSGI.

I'll address the issue in FreeIPA.

The ipautil.run() function is a helper around subprocess.Popen. The function always installs a preexec_fn in case it needs to change umask or drop priviliges. The WSGI server does not need these features.

https://github.com/freeipa/freeipa/pull/3769 should address the issue.

> I'm reducing the severity from release blocker to high and keep the ticket in pending to give Eric a change to review the commits.

Python 3.8.0 is released with the fix. It's now time to close the issue.

messages: + msg363718
stage: commit review -> resolved

resolution: fixed
stage: patch review -> commit review