Issue 32252: test_regrtest leaves a test_python_* directory in TEMPDIR

After running test_regrtest in the source tree on linux, the build/ subdirectory (i.e. test.libregrtest.main.TEMPDIR) contains a new test_python_* directory that contains a core file when the core file size is unlimited.

I did not test on 3.6.

> After running test_regrtest in the source tree on linux, the build/ subdirectory (i.e. test.libregrtest.main.TEMPDIR) contains a new test_python_* directory that contains a core file when the core file size is unlimited.

Can you please explain how to reproduce the bug? (Which commands should I type?)

What is your /proc/sys/kernel/core_pattern?

On my Fedora 27, /proc/sys/kernel/core_pattern is "|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %e".

> Can you please explain how to reproduce the bug? (Which commands should I type?)

./python -m test test_regrtest

> What is your /proc/sys/kernel/core_pattern?

$ cat /proc/sys/kernel/core_pattern
core-%e.%s
$ ulimit -c
unlimited

FWIW on archlinux (my platform) coredumps are handled by systemd and to enable coredumps the file /etc/sysctl.d/50-coredump.conf has to be updated to contain:

kernel.core_pattern=core-%e.%s

Oh it seems your Fedora 27 also uses systemd and uses the same configuration as archlinux, see https://wiki.archlinux.org/index.php/Core_dump. In that case according to this wiki your core dumps go to /var/lib/systemd/coredump and this may explain the different behaviors between your system and mine.

My setup (updating the file /etc/sysctl.d/50-coredump.conf) was the documented archlinux practice 4 years ago when systemd used to store the coredumps in log files (very annoying). This setup is still working although it is not documented in this wiki (maybe somewhere else).

FWIW I have kept the notes made when configuring archlinux at that time, they are:

disable core dumps managed by systemd-coredumpctl(1) - see also man pages for sysctl.d and sysctl
/etc/sysctl.d/50-coredump.conf                        # same file name as in /usr/lib/sysctl.d/
    kernel.core_pattern=core-%e.%s
echo "core-%e.%s" > /proc/sys/kernel/core_pattern     # or reboot

# per user
ulimit -c unlimited

I'm unable to reproduce the issue on Fedora 27:

vstinner@apu$ cat /proc/sys/kernel/core_pattern 
core-%e.%s
vstinner@apu$ ulimit -c
unlimited
vstinner@apu$ ./python -m test test_regrtest
Run tests sequentially
0:00:00 load avg: 0.70 [1/1] test_regrtest
1 test OK.

Total duration: 14 sec
Tests result: SUCCESS


While core dump works as expected:

vstinner@apu$ ./python
>>> import faulthandler
>>> faulthandler._sigsegv()
Segmentation fault (core dumped)
vstinner@apu$ ls core*
core-python.11.32456

Ah! I misunderstood the bug report. I was looking for a ENV_FAILED failure, but no, regrtest fails to remove its temporary directory but no warning is emitted in this case.

vstinner@apu$ ls -d build/test_python_*; ./python -m test test_regrtest -m test.test_regrtest.ArgsTestCase.test_crashed; ls -d build/test_python_*
ls: impossible d'accéder à 'build/test_python_*': No such file or directory

Run tests sequentially
0:00:00 load avg: 0.42 [1/1] test_regrtest
1 test OK.

Total duration: 537 ms
Tests result: SUCCESS

build/test_python_816


So running test_regrtest creates build/test_python_816/ but fails to remove it because it contains a core dump: core-python.11.816.

* No core dump must be written, Python is supposed to disable that
* regrtest must emit a warning if it fails to remove the temporary directory, maybe also fail with an error? (non-zero exit code)

What is now the content of the 'build/' subdirectory of the source tree ?

> Ah! I misunderstood the bug report. I was looking for a ENV_FAILED failure, but no, regrtest fails to remove its temporary directory but no warning is emitted in this case.

Yes :-)

> So running test_regrtest creates build/test_python_816/ but fails to remove it because it contains a core dump: core-python.11.816.

Not sure that it is because it contains a core file. Maybe it just does not remove it because the test crashed ?

> Maybe it just does not remove it because the test crashed ?

Ah, you're right :-)

> * No core dump must be written, Python is supposed to disable that

My bad. It's an obvious bug that was missed before because regrtest creates a temporary directory to run tests and then remove it.

I wrote the PR 4794 to fix faulthandler.

New changeset 48d4dd974f0c8d47c54990eedd322b96b19c60ec by Victor Stinner in branch 'master':
bpo-32252: Fix faulthandler_suppress_crash_report() (#4794)
https://github.com/python/cpython/commit/48d4dd974f0c8d47c54990eedd322b96b19c60ec

New changeset 71d8f36eb450cdbc4b5397e25f6f3f5d676aca79 by Victor Stinner (Miss Islington (bot)) in branch '3.6':
bpo-32252: Fix faulthandler_suppress_crash_report() (GH-4794) (#4795)
https://github.com/python/cpython/commit/71d8f36eb450cdbc4b5397e25f6f3f5d676aca79

This issue has been fixed in a more generic way: when using -jN, child processes reuse the temporary directory of the main process, and then main process ensures that the temporary directory is always removed:

commit 3c93153f7db5dd9b06f229e61978fd9199b3c097
Author: Victor Stinner <vstinner@redhat.com>
Date:   Tue May 14 15:49:16 2019 +0200

    bpo-36915: regrtest always remove tempdir of worker processes (GH-13312)

I close the issue. Moreover, the initial issue has been fixed.

Thanks for the report Xavier :-)

stage: patch review -> resolved