Issue 39209: Crash on REPL mode with long text copy and paste

When I copy and paste the pretty long text into REPL shell.
REPL shell is crash down with segment fault.

This issue is only reproducible on macOS,
but Linux REPL doesn't look like normal behavior.

[origin text]
<?xml version="1.0" encoding="iso-8859-1"?>
<test>
    <Users>
        <fun25>
            <limits>
                <total>0KiB</total>
                <kbps>0</kbps>
                <rps>1.3</rps>
                <connections>0</connections>
            </limits>
            <usages>
                <total>16738211KiB</total>
                <kbps>237.15</kbps>
                <rps>1.3</rps>
                <connections>0</connections>
            </usages>
            <time_to_refresh>never</time_to_refresh>
            <limit_exceeded_URL>none</limit_exceeded_URL>
        </fun25>
    </Users>
</test>

[macOS]

Python 3.9.0a2+ (heads/master:7dc72b8d4f, Jan  4 2020, 23:22:45)
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a = """<?xml version="1.0" encoding="iso-8859-1"?>
... <test>
...     <Users>
...         <fun25>
...             <limits>
...                 <total>0KiB</total>
...                 <kbps>0</kbps>
...                 <rps>1.3</rps>
...                 <connections>0</connections>
...             </limits>
...             <usages>
...                 <total>16738211KiB</total>
...                 <kbps>237.15</kbps>
...                 <rps>1.3</rps>
...                 <connections>0</connections>
...             </usages>
...             <time_to_refresh>never</time_to_refresh>
...             <limit_exceeded_URL>none</limit_exceeded_URL>
...         </fun25>
...     </Users>
... </test>
... """
Assertion failed: ((intptr_t)(int)(a - line_start) == (a - line_start)), function parsetok, file Parser/parsetok.c, line 324.
[1]    13389 abort      ./python.exe


[linux]

Python 3.9.0a2+ (heads/master-dirty:7dc72b8, Jan  4 2020, 23:22:11)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a = """<?xml version="1.0" encoding="iso-8859-1"?>
<test>
    <Users>
        <fun25>
            <limits>
                <total>0KiB</total>
                <kbps>0</kbps>
                <rps>true</rps>
                <connections>0</connections>
            </limits>
            <usages>
                <total>16738211KiB</total>
                <kbps>237.15</kbps>
                <rps>true</rps>
                <connections>0</connections>
            </usages>
            <time_to_refresh>never</time_to_refresh>
            <limit_exceeded_URL>none</limit_exceeded_URL>
        </fun25>
    </Users>
</test>
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... """
>>> a
'<?xml version="1.0" encoding="iso-8859-1"?>\n<test>\n    <Users>\n        <fun25>\n            <limits>\n                <total>0KiB</total>\n                <kbps>0</kbps>\n                <rps>true</rps>\n                <connections>0</connections>\n            </limits>\n            <usages>\n                <total>16738211KiB</total>\n                <kbps>237.15</kbps>\n                <rps>true</rps>\n                <connections>0</connections>\n            </usages>\n            <time_to_refresh>never</time_to_refresh>\n            <limit_exceeded_URL>none</limit_exceeded_URL>\n        </fun25>\n    </Users>\n</test>\n'
>>>

I bisected this to this commit:

995d9b92979768125ced4da3a56f755bcdf80f6e is the first bad commit
commit 995d9b92979768125ced4da3a56f755bcdf80f6e
Author: Anthony Sottile <asottile@umich.edu>
Date:   Sat Jan 12 20:05:13 2019 -0800

    bpo-16806: Fix `lineno` and `col_offset` for multi-line string tokens (GH-10021)

 Lib/test/test_ast.py                               | 19 +++++++++++++
 Lib/test/test_fstring.py                           | 32 +++++++++-------------
 Lib/test/test_opcodes.py                           |  2 +-
 Lib/test/test_string_literals.py                   |  8 +++---
 Misc/ACKS                                          |  1 +
 .../2018-10-20-18-05-58.bpo-16806.zr3A9N.rst       |  1 +
 Parser/parsetok.c                                  | 15 ++++++++--
 Parser/tokenizer.c                                 |  7 +++++
 Parser/tokenizer.h                                 |  5 ++++
 Python/ast.c                                       | 10 +++++--
 Python/importlib.h                                 | 16 +++++------
 Python/importlib_external.h                        | 18 ++++++------
 Python/importlib_zipimport.h                       |  8 +++---
 13 files changed, 91 insertions(+), 51 deletions(-)
 create mode 100644 Misc/NEWS.d/next/Core and Builtins/2018-10-20-18-05-58.bpo-16806.zr3A9N.rst

@pablogsal

Works correct with PR 17860 :)


Python 3.9.0a2+ (heads/pr/17860:958541d67c, Jan  6 2020, 20:45:49)
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a = """<?xml version="1.0" encoding="iso-8859-1"?>
... <test>
...     <Users>
...         <fun25>
...             <limits>
...                 <total>0KiB</total>
...                 <kbps>0</kbps>
...                 <rps>1.3</rps>
...                 <connections>0</connections>
...             </limits>
...             <usages>
...                 <total>16738211KiB</total>
...                 <kbps>237.15</kbps>
...                 <rps>1.3</rps>
...                 <connections>0</connections>
...             </usages>
...             <time_to_refresh>never</time_to_refresh>
...             <limit_exceeded_URL>none</limit_exceeded_URL>
...         </fun25>
...     </Users>
... </test>
... """
>>> a
'<?xml version="1.0" encoding="iso-8859-1"?>\n<test>\n    <Users>\n        <fun25>\n            <limits>\n                <total>0KiB</total>\n                <kbps>0</kbps>\n                <rps>1.3</rps>\n                <connections>0</connections>\n            </limits>\n            <usages>\n                <total>16738211KiB</total>\n                <kbps>237.15</kbps>\n                <rps>1.3</rps>\n                <connections>0</connections>\n            </usages>\n            <time_to_refresh>never</time_to_refresh>\n            <limit_exceeded_URL>none</limit_exceeded_URL>\n        </fun25>\n    </Users>\n</test>\n'
>>>

New changeset 5ec91f78d59d9c39b984f284e00cd04b96ddb5db by Pablo Galindo in branch 'master':
bpo-39209: Manage correctly multi-line tokens in interactive mode (GH-17860)
https://github.com/python/cpython/commit/5ec91f78d59d9c39b984f284e00cd04b96ddb5db

New changeset b2e281aaa2e4a1f24671d293dfd06b23bb052e47 by Miss Islington (bot) in branch '3.8':
bpo-39209: Manage correctly multi-line tokens in interactive mode (GH-17860)
https://github.com/python/cpython/commit/b2e281aaa2e4a1f24671d293dfd06b23bb052e47