Issue 35352: test_asyncio fails on RHEL8, or on Fedora using NEXT security policy

When compiling python3 on RHEL8 and run the tests, test_asyncio fails with 8 failures (and more on older branches).

openssl version is 1.1.1

Attaching the failures for the various branches.

Seems that 3.8 and 3.7 get the same failures, while 3.6 gets 14 instead of 8.

The strange thing is these tests are passed on our build bots.

In logs I see only ConnectionResetError. Can it be related to your configuration?

It seems I can reproduce it on Fedora as well by setting stronger crypto defaults through 'update-crypto-policies --set FUTURE'.

Repo located here: https://gitlab.com/redhat-crypto/fedora-crypto-policies/tree/master

The changes are many, but if I compare with RHEL8, the minimal changes that could affect it are:

-# DH params size: >= 1023
+# DH params size: >= 2048

-# TLS protocols: TLS >= 1.0
+# TLS protocols: TLS >= 1.2, DTLS >= 1.2

-@protocol_list = ('TLS1.3', 'TLS1.2', 'TLS1.1', 'TLS1.0', 'DTLS1.2', 'DTLS1.0');
+@protocol_list = ('TLS1.3', 'TLS1.2', 'DTLS1.2');

- $min_tls_version = 'TLS1.0';
- min_dtls_version = 'DTLS1.0';
+ $min_tls_version = 'TLS1.2';
+ $min_dtls_version = 'DTLS1.2';

# Parameter sizes
- $min_dh_size = 1023;
+ $min_dh_size = 2048;

Maybe this is the reason.
asycio test suite uses these ssl contexts: https://github.com/python/cpython/blob/master/Lib/test/test_asyncio/utils.py#L72-L92

Maybe bumping used protocol version will help to pass tests on your box.
Would you try it?

I believe I figured out the issue, at least on the master branch.

While checking the certificates used by asyncio tests within the test_asyncio folder I noticed they were quite outdated when compared to the more recent updated ones with the test/ folder, which take into account the stronger crypto defaults introduced in the latest openssl versions.

And by looking at https://github.com/python/cpython/commit/6d8c1abb003a4cb05f1ddcf0eeddeeeed513cd57#diff-a8e7dbb528601706db0f01d01332bb76 it seems that those certs are just copied from test/ within test_asyncio/. So by copying over the old certs, the tests actually pass.

The immediate workaround would be to just copy over the certs but a better approach would be to just reuse the certs within the test/ folder instead of relying on copying them over to test_asyncio/

> The strange thing is these tests are passed on our build bots.

RHEL8 has a very strict security policy by default. I'm not sure if any OS run on buildbot has a security policy as strict as RHEL8?

> Maybe bumping used protocol version will help to pass tests on your box.

I tried to tune the SSLContext in many different ways but it doesn't work. The problem comes from the .pem files.

I confirm that copying .pem files from Lib/test/ into Lib/test/test_asyncio/ does fix the issue.

> And by looking at https://github.com/python/cpython/commit/6d8c1abb003a4cb05f1ddcf0eeddeeeed513cd57#diff-a8e7dbb528601706db0f01d01332bb76 it seems that those certs are just copied from test/ within test_asyncio/. So by copying over the old certs, the tests actually pass.

In this case, I don't see the point of having two copies of the same files.

PR 10826 does the right fix: remove .pem files from Lib/test/test_asyncio/ and reuse .pem files from Lib/test/.

Also on Fedora the same set of security policies can be set as RHEL8 by utilizing 'update-crypto-policies --set NEXT'

New changeset b062ba77b617b0f89b7ea25d14cc77c991462ad4 by Victor Stinner (stratakis) in branch 'master':
bpo-35352: test_asyncio uses the certificate set from the test directory (GH-10826)
https://github.com/python/cpython/commit/b062ba77b617b0f89b7ea25d14cc77c991462ad4

IIRC two copies exist from very early development times when asyncio was not a part of Python stdlib.

> IIRC two copies exist from very early development times when asyncio was not a part of Python stdlib.

Yeah, that was my guess as well. Maybe data_file() could be simplified or replaced by support.findfile(), but I chose the easy solution (minimize changes) :-)

New changeset 7212148c95947b0fdfcb0c8e37d4357287bdb4bd by Victor Stinner in branch 'master':
bpo-35352: Cleanup test_asyncio/utils.py (GH-10831)
https://github.com/python/cpython/commit/7212148c95947b0fdfcb0c8e37d4357287bdb4bd

New changeset 38bed786a219c65d5a51c7ef4ffd97e12653a095 by Victor Stinner in branch '3.7':
[3.7] bpo-35352: test_asyncio uses the certificate set from the test directory (GH-10826) (GH-10834)
https://github.com/python/cpython/commit/38bed786a219c65d5a51c7ef4ffd97e12653a095

New changeset 02250e57c37339ea6de08ab077a307e75eef02f5 by Victor Stinner in branch '3.6':
bpo-35352: test_asyncio uses the certificate set from the test directory (GH-10826) (GH-10832)
https://github.com/python/cpython/commit/02250e57c37339ea6de08ab077a307e75eef02f5

I leave the issue open until someone validates manually that the issue os fixed in all branches with stricter security (ex: RHEL8).

I tested on Fedora 29 using:

   sudo update-crypto-policies --set NEXT

With this config, I was able to reproduce the test_asyncio failure on 3.6, 3.7 and master branches.

I confirm that the commits fixed test_asyncio in these 3 branches. Thanks Charalampos Stratakis!

resolution: fixed
stage: patch review -> resolved