Issue 23466: PEP 461: Inconsistency between str and bytes formatting of integers

PEP 461 says that all numeric bytes formatting codes will work as they do for str. In particular b"%x" % val is equivalent to ("%x" % val).encode("ascii"). But this is wrong with current implementation:

>>> '%x' % 3.14
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: %x format: an integer is required, not float
>>> b'%x' % 3.14
b'3'

The same is for %X, %o and %c.

Raising TypeError on non-integer input to %c, %o, %x, and %X was added in issue19995.

Here is a patch.

The difference between string and bytes formatting is that %c with out of the range integer raises OverflowError for str and TypeError for bytes. May be ValueError is more suitable in both cases.

Ethan?

The patch looks good to me, except of a question added on the review.

Patch looks good.

Changing the raised exceptions to ValueError  would require deprecation periods.

> Changing the raised exceptions to ValueError  would require deprecation
periods.

bytes%args is not a new feature of python 3.5? It sounds strange to
deprecate a part of a new feature.

New changeset cb96fd376baa by Serhiy Storchaka in branch 'default':
Issue #23466: %c, %o, %x, and %X in bytes formatting now raise TypeError on
https://hg.python.org/cpython/rev/cb96fd376baa

It's a new feature for 3.5 that is partly responsible for compatibility with 2.7 code.

2.7 raises Overflow error, so 3.5 should also (for out of range values -- wrong value types raise TypeError).

b'%c' is still raising a TypeError.  The error message is fine ("%c requires an integer in range(256) or a single byte") but it should be an OverflowError for backwards compatibility.

OverflowError is for platform limitations (such as the size of machine word or addressed space). When limits are well defined and platform-independent, ValueError or may be TypeError are considered as better types. It would be better to change OverflowError to ValueError or TypeError in formatting.

Here is a patch that makes bytes formatting raise an OverflowError if integer argument of %c is out of range.

> b'%c' is still raising a TypeError.  The error message is fine ("%c requires an integer in range(256) or a single byte") but it should be an OverflowError for backwards compatibility.

I don't understand why you care so much on the exact exception. It doesn't look right to me to rely on the *exact* exception raised by "%c" % arg. It's an obvious bug in the application.

Sometimes, you may want to be extra safe and catch exception while formating a message. The logging module does this. But the logging doesn't care of the exact exception, it uses a generic "except Except:" in StreamHandler.emit():

    def emit(self, record):
        try:
            msg = self.format(record)
            stream = self.stream
            stream.write(msg)
            stream.write(self.terminator)
            self.flush()
        except Exception:
            self.handleError(record)

IMO b"%c" % int must raise ValueError, not OverflowError, if the value is not in the range 0..255.

See also issue18184.

Looks good, thanks Serhiy.

New changeset 313fd1c819c5 by Serhiy Storchaka in branch 'default':
Issue #23466: Raised OverflowError if %c argument is out of range.
https://hg.python.org/cpython/rev/313fd1c819c5

the new test:

test_exc('%x', '1', TypeError, "%x format: a number is required, not str")

expects the wrong error message.

python -m unittest -v test.test_format

...
'%x' % '1' works? ... no
Unexpected  <class 'TypeError'> : '%x format: an integer is required, not str'
...

-> it's "an integer", not "a number"

New changeset 11e6986c794d by Serhiy Storchaka in branch 'default':
Issue #23466: Fixed expected error message in test_format.
https://hg.python.org/cpython/rev/11e6986c794d

Good catch, Wolfgang!

Definitely we should make test_format more unittest compatible.

messages: + msg239955

stage: resolved -> needs patch

stage: patch review