Issue 40267: Error message differs when an expression is in an fstring

There are cases, where the error message differs, when an expression is being parsed inside an fstring. For example:

>>> f'{x+}'
  File "<fstring>", line 1
    (x+)
       ^
SyntaxError: unexpected EOF while parsing
>>> (x+)
  File "<stdin>", line 1
    (x+)
       ^
SyntaxError: invalid syntax

Or with lambda definitions:

>>> f'{lambda x:x}'
  File "<fstring>", line 1
    (lambda x)
             ^
SyntaxError: unexpected EOF while parsing
>>> (lambda x)
  File "<stdin>", line 1
    (lambda x)
             ^
SyntaxError: invalid syntax

It seems that this is actually a bit bigger than this and it is not specific to f-strings. 

The error message *always* changes to `unexpected EOF while parsing` if there is an error with the last character of the input and no newline follows. For example, as made clear to me by Guido, there are even differences in error messages between exec'ing and eval'ing something:

>>> exec('x+')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1
    x+
     ^
SyntaxError: invalid syntax
>>> eval('x+')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1
    x+
     ^
SyntaxError: unexpected EOF while parsing

That's because the tokenizer adds an implicit newline to the input string, before tokenizing it, when the input comes from an exec call. (see https://github.com/python/cpython/blob/14d5331eb5e6c38be12bad421bd59ad0fac9e448/Parser/tokenizer.c#L648)

And that's not limited to a character missing, as suggested by the error message. Even when the last character itself generates a SyntaxError, the error message remains "unexpcted EOF while parsing":

>>> x+@
  File "<stdin>", line 1
    x+@
      ^
SyntaxError: invalid syntax
>>> eval('x+@')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1
    x+@
      ^
SyntaxError: unexpected EOF while parsing

Thus, a very simple fix to the specific fstring problem of this issue would be to add a newline to the string that gets parsed to the parser in fstring_compile_expr in ast.c, but I guess it'd be better to fix the tokenizer itself, if this is considered a bug.

New changeset 9a4b38f66b3e674db94e07980e1cacb39e388c73 by Lysandros Nikolaou in branch 'master':
bpo-40267: Fix message when last input character produces a SyntaxError (GH-19521)
https://github.com/python/cpython/commit/9a4b38f66b3e674db94e07980e1cacb39e388c73