Issue 29568: undefined parsing behavior with the old style string formatting

Messages (9) msg287857 - (view) Author: Jerry Dimitrov (Jerry Dimitrov) Date: 2017-02-15 14:53

Hello everyone,
This is my first bug report to the python project, so please excuse me if the metadata for this particular issue is not 100% accurate.

Today I noticed (with the help from couple of people in IRC) a strange behavior in the python string formatting functionality.

Consider the following code snippets:

```
'%(a)s %(b)' % {'a': '1', 'b': '2'}

# result:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: incomplete format
```


```
'%(a) %(b)s' % {'a': '1', 'b': '2'}

# result:
'%(b)s'

# expected result:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: incomplete format
```

It seems that there is some kind of inconsistent (undefined) behavior, during the parsing of the type character for the formatted string (tested across all major python 2.x/3.x versions).

According to the documentation for string formatting and the relevant PEPs, there is no additional info about this particular case.

I want to say thank you to Yhg1s, JustASlacker, Jerub and lz1irq for discovering this 'bug/feature' and the additional information about it.

Please let me know if this is a bug, since I am not 100% sure if this is the case.
Thanks in advance for your time!

Best Regards,
Jerry

msg287859 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * Date: 2017-02-15 15:19

In this example characters between percents are parsed as for any conversion specifier but ignored and '%(a) %' is formatted to '%'. '(a)' specifies a mapping key, ' ' is a conversion flag, the second '%' is a conversion type.

>>> '%(a) d' % {'a': 123}
' 123'
>>> '%(a) %' % {'a': 123}
'%'

This behavior is explicable but looks weird and errorprone. I'm not sure about changing behavior in maintained branches (technically this may be not a bug), but I think it is worth to make this an error in the developed branch.

msg287862 - (view) Author: R. David Murray (r.david.murray) * Date: 2017-02-15 16:13

I don't think we should change the behavior in maintenance branches.  I take it what you want to do is make it so that if the parser ends up thinking it is seeing '%%' but there is stuff between the two %s, that's an error?

msg287864 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * Date: 2017-02-15 17:05

Yes, it is what I want. Current behavior is just an artefact of the implementation. I have doubts that any other implementation of printf-like formatting has such behavior.

Bytes formatting starves from the same issue.

>>> b'% %' % {}
b'%'

But there are differences:

>>> '%12%' % ()
'%'
>>> b'%12%' % ()
b'           %'

msg287871 - (view) Author: Xiang Zhang (xiang.zhang) * Date: 2017-02-15 17:34

The documentation[1] explicitly states using % to do string format could be error-prone and recommends using str.format(). So +1 on no change in maintenance branches.

[1] https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting

msg287872 - (view) Author: Xiang Zhang (xiang.zhang) * Date: 2017-02-15 17:37

Or even not fix it in develop branch.

msg287890 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * Date: 2017-02-15 22:46

Here is a patch.

msg287916 - (view) Author: Jerry Dimitrov (Jerry Dimitrov) Date: 2017-02-16 05:48

Thanks for the documentation reference. Can we at least add this link reference [1] as a note or something like that into those documentation sections: [2] https://docs.python.org/2/library/stdtypes.html#string-formatting and  [3] https://docs.python.org/3/library/string.html#string-formatting

I couldn't fine the printf specific documentation at first, so this is why I created the report.

Regards,
Jerry

msg290276 - (view) Author: Xiang Zhang (xiang.zhang) * Date: 2017-03-24 22:43

New changeset 9f8ad3f39e0a92ed37d012b9dd237399524f0d51 by Xiang Zhang (Serhiy Storchaka) in branch 'master':
bpo-29568: Disable any characters between two percents for escaped percent "%%"  in the format string for classic string formatting. (GH-513)
https://github.com/python/cpython/commit/9f8ad3f39e0a92ed37d012b9dd237399524f0d51

History Date User Action Args 2017-03-24 22:43:13xiang.zhangsetmessages: + msg290276 2017-03-17 21:06:07larrysetpull_requests: - pull_request588 2017-03-17 21:00:33larrysetpull_requests: + pull_request588 2017-03-08 03:51:57xiang.zhangsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved 2017-03-06 11:54:59serhiy.storchakasetpull_requests: + pull_request422 2017-02-16 05:48:36Jerry Dimitrovsetmessages: + msg287916 2017-02-15 22:46:17serhiy.storchakasetfiles: + format-percent.patch
versions: - Python 2.7, Python 3.5, Python 3.6
messages: + msg287890

keywords: + patch
stage: patch review

2017-02-15 17:37:47xiang.zhangsetmessages: + msg287872 2017-02-15 17:34:21xiang.zhangsetnosy: + xiang.zhang
messages: + msg287871
2017-02-15 17:05:56serhiy.storchakasetmessages: + msg287864 2017-02-15 16:13:54r.david.murraysetnosy: + r.david.murray
messages: + msg287862
2017-02-15 15:19:37serhiy.storchakasetnosy: + larry, serhiy.storchaka, rhettinger, benjamin.peterson, ned.deily

messages: + msg287859
versions: - Python 3.3, Python 3.4

2017-02-15 14:53:02Jerry Dimitrovcreate