Issue 40612: Make traceback module's formatting of SyntaxError more similar to system formatting

The traceback module formats several edge cases of SyntaxError different than CPython's default formatting.

- There's an off-by-one error if the column offset (printed as a caret) points past the last character (example: 'a +='). The clipping is wrong and the caret points to the last character.

- If the offset is <= 0, it appears the code silently adds the length of the source text.

- The system formatting suppresses the caret if the offset is -1; the only way to suppress the caret with the traceback module is setting the offset to None (or setting the source text to None).

- The system formatting can position the caret way past the end of the source text; the traceback module clips (also see the first bullet).

I propose to make the traceback module behave the same way as the system module in all cases. I also propose to make both suppress the caret if the offset is <= 0. Finally I propose to make the system formatting limit the offset to just past the end of the source text.

I propose not to bother changing anything in 3.8 or before.

Agreed on everything.

One thing I don't really understand is if you propose to also strip trailing whitespace. Does "limit the offset to just past the end of the source text" include whitespace or not?

For example, the linked PR does not change this behavior:

➜  cpython git:(pr/20072) ./python
Python 3.9.0a6+ (heads/pr/20072:6df7662ca5, May 14 2020, 20:37:50) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> compile('1 +      ', '<string>', 'exec')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1
    1 +      
             ^
SyntaxError: invalid syntax

Should we clip just past the end of `1 +` here?

My current PR does not strip trailing whitespace. It only strips a single trailing newline (since this is usually but not always present, and we don't want to its presence to cause an extra blank line, nor do we want its absence to cause the text line and the caret line to be run together).

(And, to be clear, I don't *want* to strip trailing spaces.)

Understood.

New changeset 15bc9ab301d73f20bff47a12ef05326feb40f797 by Guido van Rossum in branch 'master':
bpo-40612: Fix SyntaxError edge cases in traceback formatting (GH-20072)
https://github.com/python/cpython/commit/15bc9ab301d73f20bff47a12ef05326feb40f797