Issue 41531: Python 3.9 regression: Literal dict with > 65535 items are one item shorter

Consider this reproducer.py:

import sys
LEN = int(sys.argv[1])

with open('big_dict.py', 'w') as f:
    print('INTS = {', file=f)
    for i in range(LEN):
        print(f'    {i}: None,', file=f)
    print('}', file=f)


import big_dict
assert len(big_dict.INTS) == LEN, len(big_dict.INTS)



And run it with any number > 65535:

$ python3.9 reproducer.py 65536
Traceback (most recent call last):
  File "/tmp/reproducer.py", line 12, in <module>
    assert len(big_dict.INTS) == LEN, len(big_dict.INTS)
AssertionError: 65535


This has not happened on python 3.8. This also happens with PYTHONOLDPARSER=1.

Also reproduces with today's git.

It appears that the 65535 key is missing regardless of the LEN value.

Bisect says 8a4cd700a7426341c2074a2b580306d2d60ec839 is the first bad commit. Considering that 0xFFFF appears a few times in that patch, that seems plausible ;)

I think what's happening is that in 'compiler_dict' (Python/compile.c), it's checking whether 'elements' has reached a maximum (0xFFFF). However, it's not doing this after incrementing; instead, it's checking before incrementing and resetting 'elements' to 0 when it should be resetting to 1. The 65535th element isn't counted.

New changeset c51db0ea40ddabaf5f771ea633b37fcf4c90a495 by Pablo Galindo in branch 'master':
bpo-41531: Fix compilation of dict literals with more than 0xFFFF elements (GH-21850)
https://github.com/python/cpython/commit/c51db0ea40ddabaf5f771ea633b37fcf4c90a495

@hroncok,

How did you discover this issue?

I'd like to clean up the code for creating dictionary literals and it might be helpful to know where such huge dictionary literals exist.
I'm guessing that they are used as lookup tables for things like Unicode code-point tables, and that they would only include constants.

@hroncok said on Twitter it was reported at https://github.com/Storyyeller/enjarify/issues/17

New changeset d64d78be20ced6ac9de58e91e69eaba184e36e9b by Miss Islington (bot) in branch '3.9':
bpo-41531: Fix compilation of dict literals with more than 0xFFFF elements (GH-21850) (GH-22107)
https://github.com/python/cpython/commit/d64d78be20ced6ac9de58e91e69eaba184e36e9b

pull_requests: + pull_request20978
stage: patch review