Issue 37032: Add CodeType.replace() method

Each type types.CodeType constructor changes, a lot of applications break. Python 3.8 added a new parameter which broke for example the Genshi project, just to name one.

I propose to add a new CodeType.replace() method, similar to datetime.datetime.replace and namedtuple._replace() for example:

$ python3
Python 3.7.3 (default, Mar 27 2019, 13:41:07) 

>>> import datetime
>>> d=datetime.datetime.now()
>>> d2=d.replace(year=2010)
>>> d, d2
(datetime.datetime(2019, 5, 24, 11, 25, 3, 839966), datetime.datetime(2010, 5, 24, 11, 25, 3, 839966))

>>> import collections
>>> Point = collections.namedtuple("Point", "x y")
>>> p=Point(1, 2)
>>> p2=p._replace(y=3)
>>> p, p2
(Point(x=1, y=2), Point(x=1, y=3))

> Python 3.8 added a new parameter which broke for example the Genshi project

Pablo fixed it in Genshi:
https://github.com/edgewall/genshi/pull/19

See also bpo-37034: "Argument Clinic omits name of keyword-only parameter on _PyArg_BadArgument() call".

Fix in Hypothesis:
https://github.com/HypothesisWorks/hypothesis/commit/8f47297fa2e19c426a42b06bb5f8bf1406b8f0f3#diff-a097452b8ffe8323641ccccac335fbf9

ipython fix:
https://github.com/ipython/ipython/commit/248128dfaabb33e922b1e36a298fd7ec0c730069

It's not like Python 3.8 is the first type in Python history that code constructor changed.

The PyCode_New() function slowly growed from 3 parameters in 1990 to 16 paramters in 2019 :-)

History of PyCode_New().


New "posonlyargcount" parameter:

commit 8c77b8cb9188165a123f2512026e3629bf03dc9b
Author: Pablo Galindo <Pablogsal@gmail.com>
Date:   Mon Apr 29 13:36:57 2019 +0100

    bpo-36540: PEP 570 -- Implementation (GH-12701)


New "kwonlyargcount" parameter, bpo-1549670:

commit 4f72a78684bbfcdc43ceeabb240ceee54706c4b0
Author: Guido van Rossum <guido@python.org>
Date:   Fri Oct 27 23:31:49 2006 +0000

    Jiwon Seo's PEP 3102 implementation.
    See SF#1549670.
    The compiler package has not yet been updated.


New "freevars" and "cellvars" parameters, PEP 227:

commit 64949cb753f206c0ca1d83f55d07afd3c179b81a
Author: Jeremy Hylton <jeremy@alum.mit.edu>
Date:   Thu Jan 25 20:06:59 2001 +0000

    PEP 227 implementation
    ...
    Include/compile.h
        Add co_freevars and co_cellvars slots to code objects.
        Update PyCode_New() to take freevars and cellvars as arguments


New "firstlineno" and "lnotab" parameters:

commit da4eb5c3b55df9b7d6957ddd0458313102f0a92e
Author: Guido van Rossum <guido@python.org>
Date:   Fri Jan 24 03:43:35 1997 +0000

    Instead of emitting SET_LINENO instructions, generate a line number
    table which is incorporated in the code object.  This way, the runtime
    overhead to keep track of line numbers is only incurred when an
    exception has to be reported.


New "stacksize" parameter:

commit 8b993a98db507cc3a75067af4c865ffc8afbada1
Author: Guido van Rossum <guido@python.org>
Date:   Fri Jan 17 21:04:03 1997 +0000

    Add co_stacksize field to codeobject structure, and stacksize argument
    to PyCode_New() argument list.  Move MAXBLOCKS constant to conpile.h.
    
    Added accurate calculation of the actual stack size needed by the
    generated code.
    
    Also commented out all fprintf statements (except for a new one to
    diagnose stack underflow, and one in #ifdef'ed out code), and added
    some new TO DO suggestions (now that the stacksize is taken of the TO
    DO list).


New "argcount", "nlocals", "flags" and "varnames" parameters:

commit 681d79aaf397850778608f35585d091fa7fe370a
Author: Guido van Rossum <guido@python.org>
Date:   Tue Jul 18 14:51:37 1995 +0000

    keyword arguments and faster calls


New "name" parameter:

commit 9bfef44d97d1ae24e03717e3d59024b44626a9de
Author: Guido van Rossum <guido@python.org>
Date:   Mon Mar 29 10:43:31 1993 +0000

    ...
    * Added function name to code object.  Print it for code and function
      objects.  THIS MAKES THE .PYC FILE FORMAT INCOMPATIBLE (the version
      number has changed accordingly)
    ...


New "filename" parameter:

commit 3f5da24ea304e674a9abbdcffc4d671e32aa70f1
Author: Guido van Rossum <guido@python.org>
Date:   Thu Dec 20 15:06:42 1990 +0000

    "Compiling" version


The very first version of the function was:

static codeobject *
newcodeobject(code, consts, names)
  object *code;
  object *consts;
  object *names;

From this commit:

commit 10dc2e8097e7a431367e72f46ddba91be93aa159
Author: Guido van Rossum <guido@python.org>
Date:   Sun Nov 18 17:27:39 1990 +0000

    Initial revision

> It's not like Python 3.8 is the first type in Python history that code constructor changed.

Oops, typo: It's not like Python 3.8 is the first *time* in Python history that the code constructor changed.

Fix in Cloud Pickle:
https://github.com/cloudpipe/cloudpickle/commit/b9dc17fc5f723ffbfc665295fafdd076907c0a93

See bpo-36886 "Failed to construct CodeType on Python-3.8.0a4" for other failures caused by the addition of the "posonlyargcount" parameter to code constructor.

New changeset a9f05d69ccbf3c75cdd604c25094282697789a62 by Victor Stinner in branch 'master':
bpo-37032: Add CodeType.replace() method (GH-13542)
https://github.com/python/cpython/commit/a9f05d69ccbf3c75cdd604c25094282697789a62

Once Python 3.8 beta1 will be released, it would be interesting to patch all projects that I had to be fixed to handle the new "posonlyargcount" to use the new "replace()" method is available.

Maybe even remove the fix for Python 3.8 alpha 4 and only use use replace() on Python 3.8 and newer.

Is there anything more on this issue?

Pablo: "Is there anything more on this issue?"

I proposed "Once Python 3.8 beta1 will be released, it would be interesting to patch all projects that I had to be fixed to handle the new "posonlyargcount" to use the new "replace()" method is available."

But that only concerns third party project,s no Python directly, so I close the issue :-)

Thanks for this great improvement!

Was about to suggest having something more futureproof when noticing this with pdbpp, found this, and submitted PRs to use it for ipython and hypothesis also.

> submitted PRs to use it for ipython and hypothesis also

Cool! Thank you for doing that ;-)

stage: patch review -> resolved