Issue 18374: ast.parse gives wrong position (col_offset) for some BinOp-s

When parsing following program:

1+2-3

ast.parse gives col_offset=3 to outermost BinOp (as shown by the attached test script). Correct col_offset would be 0 for both BinOp-s

Here's a patch that corrects col_offset for binops in both the ast module and in the compiler proper. I've incorporated Aivar's test into test_ast.py; if there are test suites for compile.c please let me know and I can add something there too.

FWIW, in my opinion the col_offsets of the two nodes should be 1 and 3, respectively (the positions of the operators).

I am not sure that changes in compile.c are needed now.

Python/Python-ast.c is generated file. And perhaps it would be better to fix CST instead of AST.

Pablo, Ivan: this old bug is able CST and AST, things that you know. You may want to have a look ;-)

Note: This bug is discussed in https://twitter.com/cfbolz/status/1146866203975139328

> in my opinion the col_offsets of the two nodes should be 1 and 3, respectively (the positions of the operators)

This would not match the documentation:

> one can get the source segment of a one-line expression node using source_line[node.col_offset : node.end_col_offset]

> in my opinion the col_offsets of the two nodes should be 1 and 3, respectively (the positions of the operators)

Such change was committed once, but it was quickly reverted because it was not well discussed and contradicts the documentation.

OK, fair enough. That means right now there is no way to find the position of the operator using the ast module at the moment, correct?

It is between the end position of the left operand and the start position of the right operand.

I think the attached patch fixes the problem. Will create a pull request soon.

New changeset 110a47c4f42cf4db88edc1876899fff8f05190fb by Ivan Levkivskyi (Carl Friedrich Bolz-Tereick) in branch 'master':
bpo-18374: fix wrong col_offset of some ast.BinOp instances (GH-14607)
https://github.com/python/cpython/commit/110a47c4f42cf4db88edc1876899fff8f05190fb

New changeset c7be35c2abd598f02a633879133caec356593241 by Miss Islington (bot) in branch '3.8':
bpo-18374: fix wrong col_offset of some ast.BinOp instances (GH-14607)
https://github.com/python/cpython/commit/c7be35c2abd598f02a633879133caec356593241

New changeset 430a9f44fe22f029ae8cfeecb46621d7e199414b by Ivan Levkivskyi (Carl Friedrich Bolz-Tereick) in branch 'master':
bpo-18374: fix tests to check the correct thing about line numbers (GH-14659)
https://github.com/python/cpython/commit/430a9f44fe22f029ae8cfeecb46621d7e199414b

New changeset 68bd9c5691c4899d21cc7fe6cce7cd22b2f5ccb0 by Ivan Levkivskyi (Miss Islington (bot)) in branch '3.8':
bpo-18374: fix tests to check the correct thing about line numbers (GH-14659) (GH-14672)
https://github.com/python/cpython/commit/68bd9c5691c4899d21cc7fe6cce7cd22b2f5ccb0

messages: + msg347370
versions: + Python 3.5, - Python 3.7, Python 3.8, Python 3.9

messages: + msg347313

assignee: serhiy.storchaka
stage: patch review -> needs patch

nosy: + sam.kimbrel
messages: + msg216696

keywords: + patch