Issue 37957: Allow regrtest to receive a file with test (and subtests) to ignore

When building Python in some uncommon platforms (I am looking at you Solaris and AIX) there are some known tests that will fail. Right now, regrtest has the ability to ignore entire tests using the -x option and to receive a filter file using the --matchfile filter.

The problem with the --matchfile option is that it receives a file with patterns to accept and when you want to ignore a couple of tests and subtests, is too cumbersome to lists ALL tests that are not the ones that you want to accept.

The problem with -x is that is not easy to ignore just a subtests that fail and the whole tests needs to be ignored.

So I suggest to add a new command line option similar to --matchfile but the other way around. Another possibility is allowing to reverse the meaning of the matchfile argument, but I find that a bit more confusing.

I'm fine with adding an --excludefile=filename option to regrtest. Do you want to try to implement it? A test can be added in test_regrtest.

Tests that fail on some platforms can be marked to skip in code.

For example:

    @unittest.skipIf(sys.platform.startswith('aix'),
                     'bpo-29972: broken test on AIX')
    def test_strcoll_with_diacritic(self):
        self.assertLess(locale.strcoll('à', 'b'), 0)

> Tests that fail on some platforms can be marked to skip in code.

We provide best-effort support for AIX, but I consider that Solaris is no longer supported, and so I would prefer to avoid code specific to Soliars in the Python source code. Except if Pablo wants to provide again best-effort support for Solaris?
https://pythondev.readthedocs.io/platforms.html

> Tests that fail on some platforms can be marked to skip in code.

One of the use cases of this feature is for people to package Python at distributions or companies and want to run the test suite without modifying the code or applying patches.

FYI in the past, Fedora and RHEL packages skipped multiple tests using downstream patches. Hopefully, we managed to reenable almost all tests.

Currently in Fedora, the python3 package use regrtest -x to skip some tests:

  # Run the upstream test suite
  # test_gdb skipped on s390x:
  #   https://bugzilla.redhat.com/show_bug.cgi?id=1678277
  # test_gdb skipped everywhere:
  #   https://bugzilla.redhat.com/show_bug.cgi?id=1734327
  # test_distutils
  #   distutils.tests.test_bdist_rpm tests fail when bootstraping the Python
  #   package: rpmbuild requires /usr/bin/pythonX.Y to be installed
  LD_LIBRARY_PATH=$ConfDir $ConfDir/python -m test.regrtest \
    -wW --slowest -j0 \
    %if %{with bootstrap}
    -x test_distutils \
    %endif
    -x test_gdb \
    %ifarch %{mips64}
    -x test_ctypes \
    %endif

https://src.fedoraproject.org/rpms/python3/blob/master/f/python3.spec

We could only skip some test methods, but it's simpler for us to skip a whole test file.

New changeset e0cd8aa70a3ce19c3d3712568940aa0cbd9aa97b by Pablo Galindo in branch 'master':
bpo-37957: Allow regrtest to receive a file with test (and subtests) to ignore (GH-16989)
https://github.com/python/cpython/commit/e0cd8aa70a3ce19c3d3712568940aa0cbd9aa97b

I'm trying to keep regrtest in sync between 3.7, 3.8 and master branches. Maybe backport this change to 3.7 and 3.8 branches once buildbots validated the change? I'm keeping them in sync mostly to make bugfixes easier, but also to make my life easier when I debug issues in 3.7 or 3.8 branch.

Ok, I reopen until the backports

New changeset 67b8a1f0f0f78ec38b8626fa9f5b2f5a55c17e15 by Victor Stinner in branch '3.8':
[3.8] Update libregrtest from master (GH-19516)
https://github.com/python/cpython/commit/67b8a1f0f0f78ec38b8626fa9f5b2f5a55c17e15

New changeset b894b669c98cc365b84cbb8d20f531f1d0686f59 by Victor Stinner in branch '3.7':
Update libregrtest from master (GH-19517)
https://github.com/python/cpython/commit/b894b669c98cc365b84cbb8d20f531f1d0686f59