Issue 24421: Race condition compiling Modules/_math.c

Once or twice I have had the math or cmath modules fail to build. There seems to be a race condition, because the _math.c file seems to be compiled twice, once for each module. Attached is a GCC wrapper that should help reproduce this fairly consistently. First, build Python normally:

$ ./configure
$ make -j2

Then force Modules/_math.c to be recompiled using my wrapper script to amplify the race condition:

$ touch Modules/_math.c
$ rm build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o
$ make -j2 CC=./slow-cc.py
running build
running build_ext
building 'cmath' extension
compiling for  cmath
./slow-cc.py [. . .]/Modules/cmathmodule.o
building 'math' extension
./slow-cc.py [. . .]/Modules/mathmodule.o
./slow-cc.py -fPIC -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/media/disk/home/proj/python/cpython/Include -I/media/disk/home/proj/python/cpython -c /media/disk/home/proj/python/cpython/Modules/_math.c -o build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o
INITIAL COMPILE
./slow-cc.py -fPIC -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/media/disk/home/proj/python/cpython/Include -I/media/disk/home/proj/python/cpython -c /media/disk/home/proj/python/cpython/Modules/_math.c -o build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o
RECOMPILE
RECOMPIILE TRUNCATING
INITIAL COMPILE FINISHED
./slow-cc.py -shared build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/mathmodule.o build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o -L/usr/local/lib -lm -o build/lib.linux-x86_64-3.6/math.cpython-36m-x86_64-linux-gnu.so
build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o: file not recognized: File truncated
collect2: error: ld returned 1 exit status
RECOMPILE FINISHED
./slow-cc.py -shared build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/cmathmodule.o build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o -L/usr/local/lib -lm -o build/lib.linux-x86_64-3.6/cmath.cpython-36m-x86_64-linux-gnu.so

Failed to build these modules:
math

Confirmed on OSX 10.10. Here's my output:

running build
running build_ext
building 'cmath' extension
./slow-cc.py -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/Users/taleinat/dev/cpython/Include -I/Users/taleinat/dev/cpython -c /Users/taleinat/dev/cpython/Modules/cmathmodule.c -o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/cmathmodule.o
building 'math' extension
./slow-cc.py -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/Users/taleinat/dev/cpython/Include -I/Users/taleinat/dev/cpython -c /Users/taleinat/dev/cpython/Modules/mathmodule.c -o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/mathmodule.o
./slow-cc.py -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/Users/taleinat/dev/cpython/Include -I/Users/taleinat/dev/cpython -c /Users/taleinat/dev/cpython/Modules/_math.c -o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o
./slow-cc.py -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/Users/taleinat/dev/cpython/Include -I/Users/taleinat/dev/cpython -c /Users/taleinat/dev/cpython/Modules/_math.c -o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o
INITIAL COMPILE
RECOMPILE
RECOMPIILE TRUNCATING
INITIAL COMPILE FINISHED
./slow-cc.py -bundle -undefined dynamic_lookup build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/mathmodule.o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o -L/usr/local/lib -o build/lib.macosx-10.10-x86_64-3.6/math.cpython-36m-darwin.so
ld: file too small (length=0) file 'build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
RECOMPILE FINISHED
./slow-cc.py -bundle -undefined dynamic_lookup build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/cmathmodule.o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o -L/usr/local/lib -o build/lib.macosx-10.10-x86_64-3.6/cmath.cpython-36m-darwin.so

I think this may have been introduced when _math.c was added as a source file of the “cmath” module in r76978 (Issue 7518). In /setup.py:585 there are two distutils.core.Extension objects, both mentioning _math.c. In the build_ext._build_extensions_parallel() method at /Lib/distutils/command/build_ext.py:449, it appears to be invoking build_extension() in two concurrent threads, once for each module, including math and cmath.

I don’t know how this code is meant to work, but it looks like either the “build_ext” class should not be creating the same object file in two different threads, or /setup.py should not be giving it the same _math.c file for two different modules.

Here is a hacky patch that adds #include "_math.c" so that it is not compiled as a separate object file. This was suggested by Mark in Issue 7518. It is far from a perfect solution, but I cannot suggest anything better without knowing more about Python’s build system.

Adding Thomas Wouters to the nosy list, since he's listed on the Experts Index under "autoconf/makefiles" as an interest area. Hopefully he can help move this forward.

Ideally, I think the solution would be to compile _math.c once, as a dependency shared between both extension modules (perhaps like a static library if necessary). Does anyone know if the build system supports this, or how to do it?

Another semi-favourable solution might be to compile _math.c twice, but send the output to separate directories, e.g. . . ./math-build/_math.o and . . ./cmath-build/_math.o. Again, I don’t know if this is practical.

Here is an alternative patch to compile _math.c once from the main makefile, before the extension modules are built. I have only tested this on Linux, building in the source directory, and from a separate directory.

I am inclined to try committing my second patch to 3.6, as the race condition keeps bugging me every now and again.

My biggest worry is that it may not work on Windows. I assume Windows does not use the Makefile, but does use setup.py. In the worst case, perhaps I could add conditional code to restore the original setup.py parameters for Windows.

New changeset 76624d47ee99 by Martin Panter in branch 'default':
Issue #24421: Compile _math.c separately to avoid race condition
https://hg.python.org/cpython/rev/76624d47ee99

New changeset a99580157968 by Martin Panter in branch '3.5':
Issue #24421: Compile _math.c separately to avoid race condition
https://hg.python.org/cpython/rev/a99580157968

New changeset d4190ed586a4 by Martin Panter in branch '2.7':
Issue #24421: Compile _math.c separately to avoid race condition
https://hg.python.org/cpython/rev/d4190ed586a4

messages: + msg256137
versions: - Python 3.4

stage: patch review

messages: + msg245429

components: + Distutils