Issue 21112: 3.4 regression: unittest.expectedFailure no longer works on TestCase subclasses

In Python 2.7 and 3.3, decorating a unittest.TestCase subclass with unittest.expectedFailure caused test discover to skip the decorated test case. Python 3.4 apparently ignores the decorator when applied to classes.

The attached file when run with Python 2.7.6 on Mac OS X produces the following output

$ python -m unittest discover
F.
======================================================================
FAIL: test_fails (test.TestControl)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/schwartz_w/Documents/testtest/test.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

----------------------------------------------------------------------
Ran 2 tests in 0.000s

FAILED (failures=1)

When run with Python 3.4.0 produces the following output.

~/Documents/testtest $ python3 -m unittest discover
F.F.
======================================================================
FAIL: test_fails (test.TestControl)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/schwartz_w/Documents/testtest/test.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

======================================================================
FAIL: test_fails (test.TestTreatment)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/schwartz_w/Documents/testtest/test.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

----------------------------------------------------------------------
Ran 4 tests in 0.001s

FAILED (failures=2)


The expectedFailure decorator when applied to a class should either skip the class or run all of its tests and check that they failed for consistency with how expectedFailure applies to test methods.

Is there a chance to get this into 3.4.1?

Berker, if you think it should go into 3.4.1, you need to set the priority to "release blocker" to bring it to the attention of the release manager.

Berker: do you consider your diff ready to go in, or is it an "early" diff (like a work-in-progress)?

> Berker: do you consider your diff ready to go in, or is it an "early"
> diff (like a work-in-progress)?

I tested my patch with test_expectedFailure.py again. The patch is not really fixes the problem described in msg215240. So, I consider it a WIP patch for now. Sorry for the noise.

Here is the output on Python 3.3:

$ python3.3 -m unittest discover -v
test_fails (test_expectedFailure.TestControl) ... FAIL
test_passes (test_expectedFailure.TestControl) ... ok

======================================================================
FAIL: test_fails (test_expectedFailure.TestControl)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/berker/projects/cpython-default/playground/test_expectedFailure.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

----------------------------------------------------------------------
Ran 2 tests in 0.001s

FAILED (failures=1)


And here is the output on Python 3.5 (with issue21112.diff patch):

$ ./../python -m unittest discover -v
test_fails (test_expectedFailure.TestControl) ... FAIL
test_passes (test_expectedFailure.TestControl) ... ok
test_fails (test_expectedFailure.TestTreatment) ... expected failure
test_passes (test_expectedFailure.TestTreatment) ... unexpected success

======================================================================
FAIL: test_fails (test_expectedFailure.TestControl)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/berker/projects/cpython-default/playground/test_expectedFailure.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

----------------------------------------------------------------------
Ran 4 tests in 0.002s

FAILED (failures=1, expected failures=1, unexpected successes=1)

Considering that I'm tagging 3.4.1 within an hour or two, and we don't have a patch yet, I'd say that this is too late to go into 3.4.1.  But I'm happy to consider it for a future 3.4.x revision.

Should this be fixed for 3.4.2?

Note: current plan for 3.4.2 is to release at the end of the month.  RC1 will be in about a week.

The patch looks good to me.

3.4.2 has been released, it seems, without this getting fixed.

3.4.3 then? Are we still happy with the patch?

> 3.4.3 then? Are we still happy with the patch?

Please see msg218727. The patch is not ready yet.

Berker, it looks like the 3.3 behaviour was buggy: TestTreatment isn't run at all! Instead, it should be run and consider failures as success.

So your patch appears more correct than what 3.3 did.

3.4.3 has been released, it seems, without this getting fixed.

3.4.4 then?

-- 
William Schwartz

On Mon, Sep 8, 2014 at 3:42 AM, Michael Foord <report@bugs.python.org>
wrote:

>
> Michael Foord added the comment:
>
> The patch looks good to me.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue21112>
> _______________________________________
>

Test looks good to me. Do you want to apply it?

New changeset a90c6d608b85 by Robert Collins in branch '3.4':
Issue #21112: Fix regression in unittest.expectedFailure on subclasses.
https://hg.python.org/cpython/rev/a90c6d608b85

New changeset ac3f1a6b1de2 by Robert Collins in branch '3.5':
Issue #21112: Fix regression in unittest.expectedFailure on subclasses.
https://hg.python.org/cpython/rev/ac3f1a6b1de2

New changeset bf789ae9bde7 by Robert Collins in branch 'default':
Issue #21112: Fix regression in unittest.expectedFailure on subclasses.
https://hg.python.org/cpython/rev/bf789ae9bde7

2.7 is sufficiently different that I haven't backported this there - plus the report said it was introduced in 3.4. If someone can show this doesn't work in 2.7 and also supply a patch, we could apply it there.