Issue 38377: test_asyncio.test_events.GetEventLoopTestsMixin.test_get_event_loop_new_process mixin breaks in the Unix environment without working /dev/shm

======================================================================
ERROR: test_get_event_loop_new_process (test.test_asyncio.test_events.TestCGetEventLoop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/test/test_asyncio/test_events.py", line 2647, in test_get_event_loop_new_process
    self.loop.run_until_complete(main()),
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/asyncio/base_events.py", line 608, in run_until_complete
    return future.result()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/test/test_asyncio/test_events.py", line 2640, in main
    pool = concurrent.futures.ProcessPoolExecutor()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/concurrent/futures/process.py", line 555, in __init__
    self._call_queue = _SafeQueue(
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/concurrent/futures/process.py", line 165, in __init__
    super().__init__(max_size, ctx=ctx)
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/queues.py", line 42, in __init__
    self._rlock = ctx.Lock()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(
OSError: [Errno 38] Function not implemented

======================================================================
ERROR: test_get_event_loop_new_process (test.test_asyncio.test_events.TestPyGetEventLoop)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/test/test_asyncio/test_events.py", line 2647, in test_get_event_loop_new_process
    self.loop.run_until_complete(main()),
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/asyncio/base_events.py", line 608, in run_until_complete
    return future.result()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/test/test_asyncio/test_events.py", line 2640, in main
    pool = concurrent.futures.ProcessPoolExecutor()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/concurrent/futures/process.py", line 555, in __init__
    self._call_queue = _SafeQueue(
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/concurrent/futures/process.py", line 165, in __init__
    super().__init__(max_size, ctx=ctx)
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/queues.py", line 42, in __init__
    self._rlock = ctx.Lock()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(
OSError: [Errno 38] Function not implemented

----------------------------------------------------------------------

The problem is that in the OpenBuildService (the build system of SUSE distributions) /dev/shm is limited to:

abuild@milic:~/rpmbuild/BUILD/Python-3.8.0rc1> ls -ld /dev/shm
drwxr-xr-x 1 root root 0 Oct  5 13:23 /dev/shm
abuild@milic:~/rpmbuild/BUILD/Python-3.8.0rc1>

Provided PR will skip test if /dev/shm doesn’t exist or it doesn’t sufficient permissions.

New changeset ddbeb2f3e02a510c5784ffd74c5e09e8c70b5881 by Victor Stinner in branch 'master':
bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() (GH-20944)
https://github.com/python/cpython/commit/ddbeb2f3e02a510c5784ffd74c5e09e8c70b5881

New changeset b1e736113484c99acb57e4acb417b91a9e58e7ff by Victor Stinner in branch '3.9':
bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() (GH-20944) (GH-20962)
https://github.com/python/cpython/commit/b1e736113484c99acb57e4acb417b91a9e58e7ff

New changeset e8056180a13b6755e4e3e5505b7bf03f79da29fb by Victor Stinner in branch '3.8':
bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() (GH-20944) (GH-20962) (GH-20966)
https://github.com/python/cpython/commit/e8056180a13b6755e4e3e5505b7bf03f79da29fb

Hum, test_resource_tracker_reused() failed twice on macOS, but it's unclear to me if it's a regression caused by my change or not.

x86-64 macOS 3.9:
https://buildbot.python.org/all/#builders/696/builds/119

FAIL: test_resource_tracker_reused (test.test_multiprocessing_forkserver.TestResourceTracker)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/test/_test_multiprocessing.py", line 5219, in test_resource_tracker_reused
    self.assertTrue(is_resource_tracker_reused)
AssertionError: False is not true

> Hum, test_resource_tracker_reused() failed twice on macOS, but it's unclear to me if it's a regression caused by my change or not.

It does appear to be caused by your change. I did a quick check with the 3.8 branch. With that PR, test_multiprocessing_forkserver fails solidly. Backing up to the previous commit, test_multiprocessing_forkserver passes.

test_multiprocessing_forkserver.test_resource_tracker_reused() failed on 3.8, 3.9 and master buildbots. It looks like a real reproducible bug. I'm not sure how my change triggered this bug.

From what I understood, the problem is that creating a SemLock has an impact on the resource tracker.

The failing unit tests checks something about the resource tracker.

The test runs a function in a child process. The function is defined in _test_multiprocessing.py. _test_multiprocessing.py calls skip_if_broken_multiprocessing_synchronize() which creates a SemLock(). At the end, the child process modifies the resource tracker because it creates a SemLock().

There are different options to fix the issue:

* Modify skip_if_broken_multiprocessing_synchronize() only only attempt to create a SemLock on Linux.
* Modify skip_if_broken_multiprocessing_synchronize() to only check if the current user is allowed to write into /dev/shm/ directory: that would allow touching the resource tracker.
* Modify the test to avoid improting _test_multiprocessing in the child process.

I don't know why the test only fails on macOS.

Note: I ignored macOS failure when I merged my PR because the macOS job of GitHub Action is failing randomly for at least 3 months and nobody attempted to fix it. So I learnt to ignore this job.

New changeset 3358da4054b9b0b045eb47dc74dee3d58bfbb1d5 by Victor Stinner in branch 'master':
bpo-38377: Fix skip_if_broken_multiprocessing_synchronize() on macOS (GH-20984)
https://github.com/python/cpython/commit/3358da4054b9b0b045eb47dc74dee3d58bfbb1d5

New changeset ec9bc2da421c456e416d991fd1fe79ac33344d9d by Miss Islington (bot) in branch '3.8':
bpo-38377: Fix skip_if_broken_multiprocessing_synchronize() on macOS (GH-20984)
https://github.com/python/cpython/commit/ec9bc2da421c456e416d991fd1fe79ac33344d9d

New changeset 1529322ca6b7b029af65aceeb04e6d47013eeeb4 by Miss Islington (bot) in branch '3.9':
bpo-38377: Fix skip_if_broken_multiprocessing_synchronize() on macOS (GH-20984)
https://github.com/python/cpython/commit/1529322ca6b7b029af65aceeb04e6d47013eeeb4

Ok, macOS buildbots are back to green, I close the issue.

stage: patch review -> resolved