Issue 29877: compileall hangs when accessing urandom even if number of workers is 1

Found on Python 3.6 on a low-resource platform (NI RoboRIO), it seems that this occurs only because the ProcessPoolExecutor is being imported. A proposed fix would only import ProcessPoolExecutor if -j > 1. Stacktrace follows:

 /usr/local/bin/python3 -m compileall -j 1 /home/lvuser/py
^CTraceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/compileall.py", line 20, in <module>
    from concurrent.futures import ProcessPoolExecutor
  File "/usr/local/lib/python3.6/concurrent/futures/__init__.py", line 17, in <module>
    from concurrent.futures.process import ProcessPoolExecutor
  File "/usr/local/lib/python3.6/concurrent/futures/process.py", line 53, in <module>
    import multiprocessing
  File "/usr/local/lib/python3.6/multiprocessing/__init__.py", line 16, in <module>
    from . import context
  File "/usr/local/lib/python3.6/multiprocessing/context.py", line 5, in <module>
    from . import process
  File "/usr/local/lib/python3.6/multiprocessing/process.py", line 311, in <module>
    _current_process = _MainProcess()
  File "/usr/local/lib/python3.6/multiprocessing/process.py", line 298, in __init__
    self._config = {'authkey': AuthenticationString(os.urandom(32)),

So calling urandom() hangs on your device because there's not enough entropy?  What OS is this running?  Modern Linux, at least, should never block in urandom() (AFAIK).

> Found on Python 3.6 on a low-resource platform (NI RoboRIO), ...

Hum, it looks like an "embedded device", likely with no entropy source.

To check if os.urandom() will block, try to call the following function in your main script:

>>> import os; os.getrandom(1, flags=os.GRND_NONBLOCK)
b'\xd1'

os.urandom() blocks at system startup until the entropy pool is initialized with 128 bytes of entropy. Once it's initialized, os.urandom() will never block again.

You may also check the number of entropy bytes available to /dev/random (Python uses /dev/urandom which is different):

$ cat /proc/sys/kernel/random/entropy_avail 
3748

For a longer explanation, see the PEP 524.

Le 19/12/2017 à 15:50, STINNER Victor a écrit :
> 
> Hum, it looks like an "embedded device", likely with no entropy source.

If it does I/O (which it probably does, being used for robotics), it
should certainly be able to extract entropy from the outside world.

If it doesn't, it's an OS implementation issue.

Of course it probably doesn't hurt to import concurrent.futures lazily.
However, people should generally not expect us to never call urandom()
when importing stdlib modules.

I'm sure that the platform (a RT linux customized by National Instruments) has issues related to urandom, as this has reared it's ugly head with other PEP 525 related issues also: https://bugs.python.org/issue29208

I'll have to try those suggestions to investigate the os.urandom issue.

In kernel logs (look at dmesg), I found two messages:

vstinner@apu$ journalctl -k|grep random
déc. 14 23:10:28 apu kernel: random: fast init done
déc. 14 23:10:32 apu kernel: random: crng init done

Sadly, I don't know which one means that urandom is initialized.

See also https://wiki.archlinux.org/index.php/Random_number_generation

Finally got around to looking at this.

# cat /proc/sys/kernel/random/entropy_avail 
183

.. ran python3 -c 'import os; os.getrandom(1, flags=os.GRND_NONBLOCK)' a few times, but didn't run into a hang. Seems like the entropy_avail is staying at about the same level?

It's worth noting that I ran into this error with last year's firmware image, and I haven't actually tested this year's firmware image for this particular issue... just that I had it on my TODO list to fix for this year. :)

I think it's still a good change to make regardless.

I am not sure PR 4856 is enhancement or bugfix.
But I think it's worth enough to backport to 3.6 and 3.7.
It caused a real UX problem.

New changeset 1d817e4c8259f49602eefe9729743f6d9d748e8d by Miss Islington (bot) (Dustin Spicuzza) in branch 'master':
bpo-29877: compileall: import ProcessPoolExecutor only when needed (GH-4856)
https://github.com/python/cpython/commit/1d817e4c8259f49602eefe9729743f6d9d748e8d

New changeset 903a3e8d67b61594c0fa17fb201769ca924b38f8 by Miss Islington (bot) in branch '3.7':
bpo-29877: compileall: import ProcessPoolExecutor only when needed (GH-4856)
https://github.com/python/cpython/commit/903a3e8d67b61594c0fa17fb201769ca924b38f8

New changeset 879f5f3d9c1f5b66e2a080c712e2323e9c03d558 by Miss Islington (bot) in branch '3.6':
bpo-29877: compileall: import ProcessPoolExecutor only when needed (GH-4856)
https://github.com/python/cpython/commit/879f5f3d9c1f5b66e2a080c712e2323e9c03d558

messages: + msg308649

type: behavior -> resource usage