Issue 32176: Zero argument super is broken in 3.6 for methods with a hacked __class__ cell

The following code works in 3.3, 3.4, and 3.5, but in 3.6 it throws  RuntimeError: super(): bad __class__ cell. 

from types import FunctionType, CodeType

def create_closure(__class__):
    return (lambda: __class__).__closure__

def new_code(c_or_f):
    '''A new code object with a __class__ cell added to freevars'''
    c = c_or_f.__code__ if isinstance(c_or_f, FunctionType) else c_or_f
    return CodeType(
        c.co_argcount, c.co_kwonlyargcount, c.co_nlocals,
        c.co_stacksize, c.co_flags, c.co_code, c.co_consts, c.co_names,
        c.co_varnames, c.co_filename, c.co_name, c.co_firstlineno,
        c.co_lnotab, c.co_freevars + ('__class__',), c.co_cellvars)

def add_foreign_method(cls, f):
    code = new_code(f.__code__)
    name = f.__name__
    defaults = f.__defaults__
    closure = (f.__closure__ or ()) + create_closure(cls)
    setattr(cls, name, FunctionType(code, globals(), name, defaults, closure))

class List(list):
    def append(self, elem):
        super().append(elem)
    def extend(self, elems):
        super().extend(elems)
def __getitem__(self, i):
    print('foreign getitem')
    return super().__getitem__(i)

add_foreign_method(List, __getitem__)

self = List([1,2,3])
self[0]

The hacked cell object using this method appears to be changed to NULL when accessed by frame.f_localsplus. I don't know C well enough to find out what's happening because nothing looks different to me in PyFrame_FastToLocalsWithError.


Also creating a closure with:
from ctypes import pythonapi, py_object

new_cell = pythonapi.PyCell_New
new_cell.argtypes = (py_object, )
new_cell.restype = py_object

doesn't solve this either.

So while CO_NOFREE is set in all versions with the example code, it appears only python 3.6 recognizes that flag and disallows the accessing of the __class__ cell. In this case the error message is bad because it implies that the __class__ cell is the wrong type. Disabling the flag when creating the code objects allows the above code to work.

Given that, I'd say the way to cleanest way to fix this would be to remove these lines from "compute_code_flags" in compile.c:

    if (!PyDict_GET_SIZE(c->u->u_freevars) &&
        !PyDict_GET_SIZE(c->u->u_cellvars)) {
        flags |= CO_NOFREE;
    }

and replace them with a check like the following in PyCode_New just after we ensure the Unicode string for the filename is ready:

    if (!PyTuple_GET_SIZE(freevars) &&
        !PyTuple_GET_SIZE(cellvars)) {
        flags |= CO_NOFREE;
    }

That way CO_NOFREE will be set only when appropriate regardless of how the code object is created, rather than relying on the caller to set it correctly.

New changeset 078f1814f1a4413a2a0fdb8cf4490ee0fc98ef34 by Nick Coghlan in branch 'master':
bpo-32176: Set CO_NOFREE in the code object constructor (GH-4675)
https://github.com/python/cpython/commit/078f1814f1a4413a2a0fdb8cf4490ee0fc98ef34

New changeset c8f32aae0aa173e122cf4c0592caec620d0d1de9 by Nick Coghlan in branch '3.6':
[3.6] bpo-32176: Set CO_NOFREE in the code object constructor (GH-4684)
https://github.com/python/cpython/commit/c8f32aae0aa173e122cf4c0592caec620d0d1de9

Thanks for the issue report! The fix will be released in 3.6.4 and 3.7.0a3 (both of which are expected to be later this month).

resolution: fixed
stage: patch review -> resolved