Issue 23972: Asyncio reuseport - Python tracker

I'm trying to create some UDP sockets for doing multicast communication. I have the working code in synchronous way and try to port it to asyncio.

One last issue is blocking for me, I'm on Mac OS X and for multicast UDP to work, the SO_REUSEPORT must be set on socket before bind. The problem is that I don't have access on socket, it's created inside asyncio method _create_connection_transport.

I've seen that SO_REUSEADDR is already set (https://github.com/gvanrossum/tulip-try3/blob/7b2d8abfce1d7ef18ef516f9b1b7032172630375/asyncio/base_events.py#L720), so maybe we could also set SO_REUSEPORT only on platforms where it's available. `if hasattr(socket, 'SO_REUSEPORT')` should works.

Or we could add an optional arguments with could be used to set some socket options, it could be more flexible that set SO_REUSEPORT.

I could provide a patch for the best solution selected.

I think we left this unfinished. I think it would be best if we added a sock parameter (like for create_server()) but also added reuse_address and reuse_port parameters, treated similarly like the one on create_server(), and added a reuse_port parameter to create_server().

Or maybe there should only be one parameter that guides both; read http://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t before you decide!

I encountered this issue too. I needed it resolved ASAP for my work so I created a loop patch that partially implements the suggestion solution by overriding the create_datagram_endpoint method. Perhaps this might be of some use to the eventual ticket resolver.

It can be found in the following gist: https://gist.github.com/claws/d4076b4b155695844d54

If this looks like it's progressing in the right direction I can try to implement it properly in `base_events.py` and submit it via a patch.

Attached is a patch that implements the suggested solution along with tests and associated doc updates. Hope this helps.

A quick search told me that "Windows only knows the SO_REUSEADDR option, there is no SO_REUSEPORT". It should be at least documented that SO_REUSEADDR is not supported on Windows.

Maybe we should raise an exception on Windows if reuse_port=True? Ignoring the parameter is a different option, but it should be well documented that the option is ignored if socket.SO_REUSEPORT is not defined.

I have updated the patch to address comments raised by haypo. 

An exception is now raised if reuse_port is explicitly used and the platform does not support SOREUSEPORT. The docs have also been updated to make it more explicit that this feature is not supported on Windows.

Hum, the latest patch was not reviewed yet :-(

Rebase patch onto current master.

I added a whole bunch of review comments. Please send a new patch!

Updates addressing review comments.

I'm adding a rebased version of Chris's v4 patch to see if I can get the code review to trigger.

New changeset 5e7e9b131904 by Guido van Rossum in branch '3.4':
Issue #23972: updates to asyncio datagram API. By Chris Laws.
https://hg.python.org/cpython/rev/5e7e9b131904

New changeset ba956289fe66 by Guido van Rossum in branch '3.5':
Issue #23972: updates to asyncio datagram API. By Chris Laws. (Merge 3.4->3.5.)
https://hg.python.org/cpython/rev/ba956289fe66

New changeset c0f1f882737c by Guido van Rossum in branch 'default':
Issue #23972: updates to asyncio datagram API. By Chris Laws. (Merge 3.5->3.6.)
https://hg.python.org/cpython/rev/c0f1f882737c

Thanks for the patch! It's live now. Closing. Watch the buildbots for me please!

I've checked the Buildbot results and observed a few errors related to this change. Specifically the issues affect Windows and Debian.

Builder AMD64 Debian root 3.x build #2772 - http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/2772
Builder AMD64 Windows7 SP1 3.x build #6808 - http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/6808

In test_create_datagram_endpoint_sock the Windows-based buildbot raises an error related to using sock.getsockname(). This issue has been observed before on Windows using Python.

Refs:
https://mail.python.org/pipermail/python-list/2015-January/683777.html
http://stackoverflow.com/questions/15638214/socket-error-invalid-argument-supplied

In this test the socket is not really used so I was trying to get away with minimal actions to set up the test. The suggested solution would be to bind the socket before using it in the test.

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('0.0.0.0', 0))


In test_create_datagram_endpoint_sockopts we can't use the (else clause at line 1310) because if SO_REUSEPORT is not defined it can not be used in a check. This affects Windows and Debian where SO_REUSEPORT is not defined.

From:
if reuseport_supported:
    self.assertTrue(
        sock.getsockopt(
            socket.SOL_SOCKET, socket.SO_REUSEPORT))
else:
    self.assertFalse(
        sock.getsockopt(
            socket.SOL_SOCKET, socket.SO_REUSEPORT))

To:

if reuseport_supported:
    self.assertTrue(
        sock.getsockopt(
            socket.SOL_SOCKET, socket.SO_REUSEPORT))

I'll submit a patch to resolve these issues later today.

This patch contains minor updates to resolve the Buildbot issues observed on the Windows and Debian platforms after the initial #23972 change set was committed.

New changeset aebbf205ef6f by Guido van Rossum in branch '3.4':
Issue #23972: Fix tests for Windows and Debian.
https://hg.python.org/cpython/rev/aebbf205ef6f

New changeset 4d643c5df2a5 by Guido van Rossum in branch '3.5':
Issue #23972: Fix tests for Windows and Debian. (Merge 3.4->3.5)
https://hg.python.org/cpython/rev/4d643c5df2a5

New changeset 3e2218a4e629 by Guido van Rossum in branch 'default':
Issue #23972: Fix tests for Windows and Debian. (Merge 3.5->3.6)
https://hg.python.org/cpython/rev/3e2218a4e629

messages: + msg252379

assignee: gvanrossum
resolution: fixed

messages: + msg252223

messages: + msg252189

messages: + msg251837

messages: + msg249000