PEP 605: A rolling feature release stream for CPython

(Emphasis mine.) Maybe we could acknowledge this in the text, then? That way future readers can look at it and go “okay, the authors weren’t 110% sure this idea was perfect, so it’s okay to try something else” without having to track us down in our nursing homes

1 Like

I’ve hit the merge button on that PR now. If the rendered version of the PEP has pictures showing example timelines and support period overlaps, then it’s showing the version of the proposal I’m referring to in the quote above

Relative to the version originally posted at the start of this thread, the substantive change is the one to actively encourage the publication of pre-built binary wheels for a release series as soon as the first pre-release is made available. That change in turn led to the reintroduction of routine alpha releases, specifically to mark pre-releases that include breaking changes in the full CPython C ABI.

The rest of the changes in the update are editorial ones that aim to make the PEP easier to read (adding pictures similar to those in PEP 602, presenting the example timeline earlier in the text, reducing the length of the abstract, etc).

(Note: the in-thread proposal has been updated to match the most recently published version, with a pointer down to this comment to indicate which parts of the thread related to the older version. With a mailing list, I would have just started a new thread for the new version, but that doesn’t seem likely to work as well on a web forum, since old topics don’t decay as quickly).

2 Likes

On re-reading the design discussion section on the alpha/beta naming scheme, I realised I didn’t have this metadata declaration quite right yet, due to the fact that all alpha releases sort as earlier than beta releases.

Instead, I think the required metadata to express the desired constraint would be:

Python-Requires: >= "3.9.0b6"; python_version == "3.9" and python_full_version != "3.9.0a5"

That carves out an exception to allow installation on the alpha build that defines the relevant target ABI, rather than incorrectly allowing installation on all beta builds (even those that preceded the relevant alpha build).

Once the 3.9 series was stable, then those qualifiers could be dropped from the metadata.

It’s weird that a new version stream always starts with a “alpha 1” release and then switches to “beta 2”. While I understand the motivation, it breaks users’ expectation wrt. to alpha and beta releases (usually you get all alpha releases clustered together at one point of the development process, then a sequence of beta releases start from “beta 1”).

I would still recommend that this proposal drops alpha releases altogether, for clarity and ease of adoption.

Otherwise, I’m impressed that this seems to be the first release model proposal that successfully addresses the needs of various groups in the ecosystem without presenting blocking barriers for any of these groups.

4 Likes

I think there’s agreement that noting the abi breaks in some way is important so that the wider ecosystem does not have to repackage the world every two months. IMO this should be obvious straight from the version numbers/tags.

However, I keep circling back that the names “alpha” and “beta” have a lot of historical baggage (and strong connotations from other software projects). In particular, I don’t think the following will hold true in practice:

IMO, these conventions are very hard to replace, and trying to assign a different meaning to the suffix “b” will not stop >95% people reading a version label 3.9.0b3 as a beta version.

@ncoghlan, @steve.dower, why not extend the the pre-release tag “p” (already proposed as a SOABI-marker in this PEP) to be a fully-fledged pre-release marker (cf. PEP 440)? Replacing the usages of “beta” in this PEP with “pre-release” and the “b”-suffix with “p” would IMO solve several problems, the most crucial of which I regard to be that there’s no previous expectations what 3.9.0p3 means (re: adoption of beta releases), and that the naming would explicitly signal that the “p”-releases are considered “above-beta-quality”.

As such, it would even be possible to also “upgrade” the alphas in this PEP to betas (if the idea is to keep the number of possible tags in the rolling stream as low as possible), and denote the ABI-breaks by a “b” (“breaking?”).

There seems to be a fundamental disconnect here, where folks seem to think Steve and I want everyone to start using the rolling releases. We don’t - we only want folks that are thoroughly dissatisfied with the slower full release cadence to use them.

Users in that position aren’t going to be deterred by the beta label, or the fact that alphas & betas potentially become intermingled: they’ll only be deterred by a genuinely bad developer experience on the rolling release stream (which the PEP aims to avoid).

If folks with the expectation “this is exactly like using one of the stable releases” were to start using the rolling release stream, that would be a communication failure, rather than a success.

The real test of suitability would probably be that first mid-stream alpha release, where the ABI gets changed, and at least some third party packages need to be rebuilt due to segfaults. If folks get bitten by that, I assume at least some of them would say “This is not for me” and go back to the stable release series. But a lot would say “This is what I signed up for”, install whatever library updates were needed, and continue on.

Globally, my guess is that the latter group would number somewhere in the thousands (somewhere between 0.1% and 1% of our user numbers), which should be sufficient for more robust pre-release feedback. It wouldn’t be the millions that use the stable releases, but that’s a large part of the point: encouraging a larger group of self-selected early adopters to contribute to improving API designs before we lock them down in a stable release for the rest of our multiple-orders-of-magnitude larger overall user community.

3 Likes

On that front, perhaps it would make sense for me to mock up some example alpha, beta and release candidate announcements, based on the 3.8.0 pre-release cycle?

If the PEP were to be adopted, I would expect those announcements, along with a What’s New section for them to link to, would be our main tool for communicating the process change outside the core development group.

(trawling the “Include/” commit history should also give me some stats on how many potential ABI changes there actually were during 3.8.0 development)

Mostly because of PEP 440 (and the related PEP 508): if we introduce a new release type into the version number itself, we’d have to update the various packaging interoperability specs to cope with it. In particular, PEP 508 defines the “python_full_version” marker (3.9.0a1, etc), and then refers back to PEP 440 for the rules on how to order Python versions.

The main reason I could see us some day tweaking the formal naming is if everything is going wonderfully well with the rolling releases, but there are some environments that are operationally capable of handling the rolling release stream, but have an explicit policy requirement that says “No beta software”.

That kind of context is essentially the only time I could see the difference between “Formally defined as ‘binary compatible’ release, informally known as ‘beta’ releases” and “Formally defined as ‘beta’ releases” actually mattering.

We’re a long way from needing to worry about that, though - even if PEP 605 were to be adopted for 3.9, changing the rolling release type names would be a topic for 3.10 at the earliest.

Example text in this PR: https://github.com/python/peps/pull/1200

I ended up using completely hypothetical ABI break examples, as finding them without a clear marker in the commit log is tricky. I did use the real 3.8.0a1 and 3.8.0rc1 announcements as my base templates, though.

I don’t know. But other people still occasionnally look at how Python’s release process works, and they may even sometimes, out of curiosity, decide to try out an alpha or beta version. So these other people can end up confused as well.

So just to be clear, then, for the vast majority of end users you intend this proposal to be a change from an 18-month release cycle to a 2 year release cycle?

Given that your intention is for the “rolling feature release” stream to be only applicable to a tiny minority of Python users, I think it would be clearer if you presented this proposal as a 2-year release cycle with an improved process for those people who wanted to be involved in test releases.

Unless I’m still misunderstanding the intention here…

To give my personal view, as an end user, I’m not “thoroughly dissatisfied” with the longer release cycle, but I dislike it. So I guess I’m in the category of people who prefer the current situation (or the annual cycle Łukasz is proposing) but would have to put up with a longer release cycle if this proposal was accepted. So still -1, I guess…

1 Like

From my understanding, it’s not only targeted at those who are interested in test releases. In theory, it would eventually become appealing to anyone who wants to have early features and doesn’t have a critical need for stability. OTH, the current alpha and beta versions are pretty much exclusively appealing to testers and library maintainers.

Also, when Nick was referring to 0.1 to 1% of the user base, I think he was talking about the initial percentages for the first mid-stream alpha release, not the long term usage rates. It will likely remain a minority in comparison to the stable releases, but not to that degree.

I would rephrase that last part to “is in control of their own stability”, as in:

• they can choose precisely which version of Python to use
• they know that it won’t change unexpectedly
• they can validate their entire app/deployment to confirm that nothing has broken

For example, a web app in a Docker container would “qualify”, whereas an app relying on the system maintained Python provided by a cloud hosting service would not.

1 Like

So you are saying we may interleave alphas and betas?

Having to “find” ways to handle this sort of thing seems a little worrisome to lean on. Personally, I’m going to assume we won’t and view the PEP from that perspective.

2 Likes

Some are listed in the PEP - see How would users of the rolling pre-freeze releases detect API changes - but it’s deliberately set as “examples of how to achieve feature checks without version checks” rather than “here is our specific, concrete plan for the unforeseeable future”.

From a technical point of view with regards to how version numbers are compared, yes.

But better to think of it as one stream of releases with ABI-breaking releases being explicitly called out with a distinct label.

ABI-breaking or ABI-augmenting? It doesn’t seem very clear.

Nick has thought about this aspect more than me, but the intent is to communicate that “this release will require non-stable ABI extension modules to be recompiled, and existing platform-specific wheels will fail to load”. So I’d say breaking, as an augmentation that doesn’t break anything wouldn’t require this warning.

And yes, it’s changing the definition from what we currently have. The first RC becomes the point where everything is frozen, but then we plan on having it in RC for two months and would no doubt (informally) discourage major ABI changes in the last couple of pre-RC releases.

1 Like

Ah, nope. We’re actually proposing coming down really hard on the stable ABI and disallowing changes to it after something is added to any pre-release. So if something shows up in say X.Yb7, it’s guaranteed stable in X.Y. Which really raises the bar on the stable ABI, but treating it that seriously is the only way to help library developers trust that they can rely on it. And if they do, they get out completely from the p ABI flag as well, and aren’t forced to rebuild for the RC release.

But it’s an added burden on the core dev side, for sure. (Of course, it’s very easy to avoid by not adding your new API to the stable ABI until you’ve tested it in the CPython API and know it’s ready )