Issue 12546: builtin __format__ methods cannot fill with \x00 char

This gives me "foo   " instead of expected "foo\x00\x00\x00" :

"{0:\x00<6}".format('foo')

\x00 is used as a flag internally meaning "use the default fill character". That's clearly a bug. I'll look at fixing it.

I think there are other places in the built in __format__ functions where special values are used instead of flags. I'll review those as well.

Thanks for the report!

This patch removes the special meaning for \x00 and defines the default padding character (' ') in parse_internal_render_format_spec. Test included. Maybe the default padding character should be defined elsewhere?

Oops, sorry. Above patch was overly buggy. Please just ignore it.

Here's the patch. Same rationale as above (removed the special meaning of '\x00', default specified in parse_internal_render_format_spec). Sorry about the mess again!

Patch looks good at first glance. I'll review it some more today and commit it. Thanks!

I finally got around to reviewing the patch. A couple of comments:

1. There should be some tests for str.__format__, not just str.format. This is really a bug with str.__format__, after all. I can add those.

2. The bigger issue is that the other built in formatters have this same problem.

>>> format(3, '\x00<6')
'3     '
>>> format(3., '\x00<6')
'3.0   '
>>> format('3', '\x00<6')
'3\x00\x00\x00\x00\x00'
>>> format(3+1j, '\x00<6')
'(3+1j)'
[38654 refs]
>>> format(3+1j, '\x00<10')
'(3+1j)    '

I think the fix is basically the same as str.__format__ (but in format_int_or_long_internal, format_float_internal, and format_complex_internal). I tried that and it worked, but I haven't had time to write tests for them. If you (Davide) can do that, great. Otherwise I'll try and grab some time this week.

Changing the subject to match the wider scope of the problem.

#17705 has been closed as a duplicate of this issue.

The patch looks good to me.

New changeset 520ce42ba2b8 by Eric V. Smith in branch '2.7':
Issue #12546: Allow \x00 as a fill character for builtin type __format__ methods.
http://hg.python.org/cpython/rev/520ce42ba2b8

New changeset 7c484551bce1 by Eric V. Smith in branch '3.4':
Issue #12546: Allow \x00 as a fill character for builtin type __format__ methods.
http://hg.python.org/cpython/rev/7c484551bce1

New changeset bd90e68dc81f by Eric V. Smith in branch 'default':
Closes issue #12546: Allow \x00 as a fill character for builtin type __format__ methods.
http://hg.python.org/cpython/rev/bd90e68dc81f

Fixed in 2.7, 3.4, 3.5.

I don't understand why it works with "<", "=" or ">":

>>> "{0:\x00<6d}".format(123)
'123\x00\x00\x00'

But not without:

>>> "{0:\x006d}".format(123)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Invalid format specifier

Compare it to:

>>> "{0:6d}".format(123)
'   123'
>>> "{0:06d}".format(123)
'000123'

For int, the spec is:
[[fill]align][sign][#][0][width][,][.precision][type]

So, for "06d", "0" is matched as the literal 0, "6" is matched as width, and "d" is matched as type.

For "\x00<6d", "\x00" is matched as fill, "<" as align, "6" as width, and "d" as type.

For "\x006d", there's no align. So "\x00" cannot match as fill. "\x00" doesn't match anything else, so it's an invalid format specifier, thus the exception.

stage: patch review -> resolved

messages: + msg140238

nosy: + davide.rizzo
messages: + msg140233

keywords: + patch

assignee: eric.smith
components: + Interpreter Core, - Library (Lib)
stage: needs patch