Issue 39439: Windows Multiprocessing in Virtualenv: sys.prefix is incorrect

I upgraded from Python 3.7.1 to 3.7.6 and began noticing a behavior that was breaking my code. My code detects if it's running in a virtualenv. This check worked in 3.7.1 but is broken in 3.7.6.

From the documentation, sys.prefix and sys.exec_prefix should point to the virtualenv when one is active. However, I'm seeing that both of these constants are pointing to the system installation directory and not my virtualenv when I am in a multiprocessing child. Here is an example output of a test application running in 3.7.6 (I've attached the test script to this ticket):

Parent process
=============================================
sys.prefix:           C:\Users\user\project\venv
sys.exec_prefix:      C:\Users\user\project\venv
sys.base_prefix:      C:\Program Files\Python37
sys.base_exec_prefix: C:\Program Files\Python37
=============================================

Subprocess
=============================================
sys.prefix:           C:\Program Files\Python37
sys.exec_prefix:      C:\Program Files\Python37
sys.base_prefix:      C:\Program Files\Python37
sys.base_exec_prefix: C:\Program Files\Python37
=============================================


I would expect that sys.prefix and sys.exec_prefix to be identical in the parent and child process.

I verified that this behavior is present in 3.7.5, 3.7.6, and 3.8.1. I am on a Windows 10 x64 system.

Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)] on win32

This is a duplicate of issue 38092. The required change to fix this is very simple, if you want to work on a PR.

OK, that makes sense.

For 3.7, I can create a PR for that corrects the order of arguments passed into _winapi.CreateProcess

For 3.8 / master, the problem appears to be that the check in popen_spawn_win32.py to set the subprocess env is failing because sys.executable != spawn.get_executable() -- spawn.get_executable() is returning sys._base_executable. So, can you confirm that the fix is to just change spawn.get_executable() to return sys.executable, like it was prior to the PR mentioned in the other ticket?

> can you confirm that the fix is to just change spawn.get_executable() 
> to return sys.executable, like it was prior to the PR mentioned in 
> the other ticket?

Yes, in spawn.py, change `_python_exe = sys._base_executable` to `_python_exe = sys.executable`. This is the value that gets returned by spawn.get_executable().

This will restore the operation of the following code in Popen.__init__ in popen_spawn_win32.py:

        python_exe = spawn.get_executable()

        # bpo-35797: When running in a venv, we bypass the redirect
        # executor and launch our base Python.
        if WINENV and _path_eq(python_exe, sys.executable):
            python_exe = sys._base_executable
            env = os.environ.copy()
            env["__PYVENV_LAUNCHER__"] = sys.executable

This code switches to running the base executable in a virtual environment, which is required in order to support duplicating handles to the child or stealing handles from the parent. And the startup code in the child process is made aware of virtual environment via the "__PYVENV_LAUNCHER__" environment variable.

New changeset 0be3246d4f9c8eddcd55491901d95b09fe163f15 by Steve Dower (Adam Meily) in branch 'master':
bpo-39439: Fix multiprocessing spawn path in a venv on Windows (GH-18158)
https://github.com/python/cpython/commit/0be3246d4f9c8eddcd55491901d95b09fe163f15

New changeset dbb37aac1428ddcba784d9ddc77c1708c391aa80 by Miss Islington (bot) in branch '3.8':
bpo-39439: Fix multiprocessing spawn path in a venv on Windows (GH-18158)
https://github.com/python/cpython/commit/dbb37aac1428ddcba784d9ddc77c1708c391aa80

superseder: environment variables not passed correctly using new virtualenv launching in windows and python3.7+
nosy: + eryksun

messages: + msg360582
type: behavior
resolution: duplicate
stage: resolved