Flask-SocketIO needs your help!

Posted by
on under

Some of you know that for the last few weeks I have been quietly but steadily working on a significant new release of Flask-SocketIO that will be labeled 1.0, and that is practically a complete rewrite. Given that this is a fairly popular extension, I would like to ask existing users to test it and provide feedback before it is officially released.

What Is Changing?

A lot of things are changing under the hood in 1.0, and inevitably a few things are also changing in the public interface. However, I expect most existing applications will require little or no changes to run on the 1.0 release.

The big change is in the dependencies. Flask-SocketIO has depended on gevent-socketio to provide the Socket.IO server, and this was problematic because this project hasn't been updated in a long time, forcing users to work with a very old Javascript client. I have dropped gevent-socketio and replaced it with two new packages written by myself, called python-engineio and python-socketio. These packages provide the Socket.IO server-side functionality, now using current versions of the Engine.IO and Socket.IO protocols.

The heavy dependency on gevent has also been removed. In this new version you can choose to use eventlet, gevent or a plain old WSGI web server such as Werkzeug. Out of these three options, eventlet is currently the most complete implementation, with support for the HTTP long-polling and WebSocket transports. The other options only support long-polling at this time.

As a nice side effect of these changes, I can now offer support for Python 3, and since eventlet is very stable on both Python 2 and Python 3, that means that you can use the current Socket.IO client, with the WebSocket transport, on Python 2.7, 3.3 or 3.4.

Here is a more complete list of changes in version 1.0, extracted from the documentation:

  • Release 1.0 drops support for Python 2.6, and adds support for Python 3.3, Python 3.4, and pypy.
  • Releases 0.x required an old version of the Socket.IO Javascript client. Starting with release 1.0, the current releases of Socket.IO are supported (1.3.5 and 1.3.6 have been tested to work).
  • The 0.x releases depended on gevent, gevent-socketio and gevent-websocket. In release 1.0 gevent-socketio and gevent-websocket are not used anymore, and gevent is one of three options for backend web server. The other two options are eventlet, and any regular multi-threaded WSGI server, including Flask's development web server.
  • The Socket.IO server options have changed in release 1.0. They can be provided in the SocketIO constructor, or in the run() call. The options provided in these two are merged before they are used.
  • The 0.x releases exposed the gevent-socketio connection as request.namespace. In release 1.0 this is not available anymore. The request object defines request.namespace as the name of the namespace being handled, and adds request.sid, defined as the unique session ID for the client connection, and request.event, which contains the event name and arguments.
  • To get the list of rooms a client was in the 0.x release required the application to use a private structure of gevent-socketio, with the expression request.namespace.rooms. This is not available in release 1.0, which includes a proper rooms() function.
  • The recommended "trick" to send a message to an individual client was to put each client in a separate room, then address messages to the desired room. This was formalized in release 1.0, where clients are assigned a room automatically when they connect.
  • The 'connect' event for the global namespace did not fire on releases prior to 1.0, due to a problem in gevent-socketio. This has been fixed and now this event fires as expected.
  • Support for client-side callbacks was introduced in release 1.0.

Testers Wanted

If you use Flask-SocketIO, I would appreciate it if you try your application with my current working 1.0 release and provide feedback. I'm particularly interested to find out if you had to make lots of changes to your application to make it work with this version. I made every effort to minimize the changes to the public API, but I did have to make some changes.

How To Install Flask-SocketIO 1.0

I have pushed my current 1.0 version to PyPI as release 1.0a1 (Edit: current version is 1.0b1). Because it is a development version, you have to explicitly request this version to get it:

$ pip install flask-socketio==1.0b1

Or if you are upgrading an existing virtualenv:

$ pip install --upgrade flask-socketio==1.0b1

You can find the example application here: https://github.com/miguelgrinberg/Flask-SocketIO/tree/v1.0/example

If you prefer to install from GitHub, I have pushed this version to the official repository under branch v1.0.

How To Provide Feedback

You can post your comments below, or find me on Twitter or elsewhere. Just contact me in the way that is easier for you.

I appreciate your feedback!

Miguel

Become a Patron!

Hello, and thank you for visiting my blog! If you enjoyed this article, please consider supporting my work on this blog on Patreon!

59 comments
  • #1 Cesco said

    Any Python client avaible for the extension? I have been using the extension for quite a while and have never been able to find a python client option that will contemplate both namespaces and events.

  • #2 Mathieu Virbel said

    Count me in. Great job!

  • #3 Miguel Grinberg said

    @Cesco: probably not, I actually did not look for one, since the normal use case is a client that runs as a single-page app in the browser.

  • #4 Keno said

    @Cesco: I have encountered use cases for a python client, and https://pypi.python.org/pypi/socketIO-client does a decent job. You'd have to stick with version 0.5.6 for flask-socketio 0.x.x since it uses gevent-socketio

    @Miguel: I had been successfully running a setup consisting of 0.x.x version on one port, and a separate web application on another port, but after upgrading to 1.0.x, I'm getting a cross-site error when my frontend clients try to connect. Do I need any additional settings for CORS support?

    Note:
    I'm running the socketio server under gunicorn with an eventlet worker (had to change to this since the gevent dependency has been dropped), and reverse-proxying via apache (2.4+ now has support for web sockets)

    Thanks!

  • #5 Miguel Grinberg said

    @Keno: I'll retest cross-site, the default should be that all origins are allowed. Is this an error in regular HTTP requests or in the websocket request?

  • #6 Keno said

    The websocket request

  • #7 Miguel Grinberg said

    @Keno: please upgrade package python-engineio to version 0.5.1 and let me know if that fixes cross-site. Thanks!

  • #8 Keno said

    The cross-site error still persists after upgrading python-engineio to 0.5.1

  • #9 Miguel Grinberg said

    @Keno: Hmm. I was able to make cross-site websocket work with this, and I did reproduce the failure. Could you give me the Javascript console output for the error please? And also, what browser are you testing with? Maybe there is some oddity I'm not aware of in some browser. Thanks.

  • #10 Keno said

    "XMLHttpRequest cannot load https://localhost:4005/socket.io/1/?t=1439859749992. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'https://localhost:8443' is therefore not allowed access. The response had HTTP status code 404" - javascript

    I'm getting the above error for both Safari 8.0.2 and Chrome 44.0.x.

    I'm still trying to find out if it's a problem with my apache config, but the exact config I'm using seemed to work with flask-socketio 0.6.0

  • #11 Cesco said

    @Keno I was considering using that library, but as of right now I rather upgrade my projects to Python3 with this new version of @Miguel extension.

    Plus, I was trying to make a smart home automation on/off swith to my house that would listen to websockets events. But I asked the same question to @Miguel on the old post and he made me realize all I need was some regular sockets. I was thinking about using a Raspi to do the job, but as of right now I wrote all the code using an Arduino. And it just works perfectly. I am just concerned about security of this sockets. Would not it be too easy to read using Wireshark or something? Thanks @Miguel you keep teaching me a lot.

  • #12 @Cesco said

    I am trying to use the extension with an app factory inside a blueprint. But i am getting this weird error

    raise RuntimeError('Cannot associate a SocketIO instance with '
    RuntimeError: Cannot associate a SocketIO instance with more than one application

    I am not running any other instances. I even shutdown the computer to make sure. It seems that I am making something wrong between the init_app() and the run(). I have created this gist with the portion of code I thought it was relevant. https://gist.github.com/cescoferraro/dd5318a2085fb10f5254

  • #13 Miguel Grinberg said

    @Cesco: I think you are calling init() or init_app() more than once on this extension. Can you check by adding a print statement or looking in a debugger?

  • #14 Miguel Grinberg said

    @Keno: you have apache as a reverse proxy in the middle? Can you try without apache and tell me if that works correctly?

  • #15 Max said

    I have just upgraded to test it in my app.
    I had to rewrite some code as room param in emit() is not inside **kwargs.
    I had:
    socketio.send('update_participants',
    room='conference-%s' % conference.id)
    Now I have:
    socketio.emit('update_participants',{
    'room':'conference-%s' % conference.id})

    So I tried just to replace .emit to send as send() still accepts room as separate param.
    Ooops. All stopped working.
    Aaa. send() is expecting data as param not **kwargs..
    So anyway I cannot quickly upgrade just renaming emit to send.
    So I had to roll back and include room param in emit() inside dict.

    P.S. I don't see any page hits in console any more. How can I get it?

    Thanks!

  • #16 Max said

    And getting many tracebacks on client disconnects (F5 on browser) - http://pastebin.com/1ryzxC8B

  • #17 Max said

    In eventlet mode it does not run:

    Traceback (most recent call last):
    File "./run.py", line 14, in <module>
    engineio_logger=False,
    File "/home/max/tmp-work/astconfman/env/local/lib/python2.7/site-packages/flask_socketio/init.py", line 259, in run
    log_output=log_output, **kwargs)
    TypeError: server() got multiple values for keyword argument 'log_output'

    This is how I call it:
    if name=='main':
    socketio.run(app,
    host=app.config['LISTEN_ADDRESS'],
    port=app.config['LISTEN_PORT'],
    async_mode='eventlet',
    log_output=True,
    use_reloader=True,
    logger=False,
    engineio_logger=False,
    )

    Also neither gevent mode works:

    File "/home/max/tmp-work/astconfman/env/local/lib/python2.7/site-packages/engineio/socket.py", line 74, in handle_get_request
    start_response)
    File "/home/max/tmp-work/astconfman/env/local/lib/python2.7/site-packages/engineio/socket.py", line 109, in _upgrade_websocket
    return ws(environ, start_response)
    File "/home/max/tmp-work/astconfman/env/local/lib/python2.7/site-packages/engineio/async_gevent.py", line 34, in call
    self._sock = environ["wsgi.websocket"]
    KeyError: 'wsgi.websocket'
    {'GATEWAY_INTERFACE': 'CGI/1.1',
    'HTTP_CACHE_CONTROL': 'no-cache',
    'HTTP_CONNECTION': 'Upgrade',
    'HTTP_COOKIE': 'io=58865443ce8642619f52f8fcce4a45ef; session_id=1904c055261032be14c24d547e3f460ca11a4b6d; session=eyJjc3JmX3Rva2VuIjp7IiBiIjoiTnpBNE1qWXdPVE5tTURnMFky,
    'HTTP_HOST': 'localhost:5001',
    'HTTP_ORIGIN': 'http://localhost:5001',
    'HTTP_PRAGMA': 'no-cache',
    'HTTP_SEC_WEBSOCKET_EXTENSIONS': 'permessage-deflate; client_max_window_bits, x-webkit-deflate-frame',
    'HTTP_SEC_WEBSOCKET_KEY': 'J10nsEpjj7mrXQs9RIeJ2Q==',
    'HTTP_SEC_WEBSOCKET_VERSION': '13',
    'HTTP_UPGRADE': 'websocket',
    'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36',
    'PATH_INFO': '/socket.io/',
    'QUERY_STRING': 'EIO=3&transport=websocket&sid=58865443ce8642619f52f8fcce4a45ef',
    'REMOTE_ADDR': '127.0.0.1',
    'REMOTE_PORT': '59390',
    'REQUEST_METHOD': 'GET',
    'SCRIPT_NAME': '',
    'SERVER_NAME': 'explorer',
    'SERVER_PORT': '5001',
    'SERVER_PROTOCOL': 'HTTP/1.1',
    'SERVER_SOFTWARE': 'gevent/1.0 Python/2.7',
    'wsgi.errors': <open file '\<stderr>', mode 'w' at 0x7faa244651e0>,
    'wsgi.input': <gevent.pywsgi.Input object at 0x3d60090>,
    'wsgi.multiprocess': False,
    'wsgi.multithread': False,
    'wsgi.run_once': False,
    'wsgi.url_scheme': 'http',
    'wsgi.version': (1, 0)} failed with KeyError

    And in threading mode neither log_output or use_reloader works as DEBUG is False.
    But when DEBUG = True it does not work either because you don't pass these options here:

    app.run(host=host, port=port, threaded=True,
    use_reloader=use_reloader)

  • #18 Max said

    I managed to run it in eventlet mode only after adding kwargs.pop('async_mode'):

     if not test_mode:
            if self.server.eio.async_mode == 'threading':
                app.run(host=host, port=port, threaded=True)
            elif self.server.eio.async_mode == 'eventlet':
                import eventlet
                kwargs.pop('async_mode')
                eventlet.wsgi.server(eventlet.listen((host, port)), app,
                                     log_output=log_output, **kwargs)
    

    Without it I got:

    Traceback (most recent call last):
    File "./run.py", line 13, in <module>
    async_mode='eventlet',
    File "/home/max/tmp-work/astconfman/env/local/lib/python2.7/site-packages/flask_socketio/init.py", line 258, in run
    log_output=log_output, **kwargs)
    TypeError: server() got an unexpected keyword argument 'async_mode'

  • #19 Max said

    So digging the code I understood that some options must only be passed to init like:
    socketio = SocketIO(app, logger=False, engineio_logger=False)
    because later in run() you update self.server_options from kwargs but pass kwargs later to eventlet.wsgi.server(eventlet.listen((host, port)), app,
    log_output=log_output, **kwargs)
    And it does not understand it.

    So for now it works for me except the fact I lost all debug output when DEBUG=True and threading is used.
    Do you have any idea why?

  • #20 Miguel Grinberg said

    @Max: thanks for the detailed information. The debug output problem has been reported already, I am looking into that problem and hope to have it fixed in a new beta release in the next few days.

    Now, regarding the other problems you've had:

    • The socketio.send vs. socketio.emit problem: I'm not sure about this one. Both emit and send take the room argument, why didn't the original call work? Can you give me more details on that?

    • The broken pipe errors when a client disconnects (such as when you press F5). This I've seen when using eventlet. I consider those an annoyance more than a bug, it appears some code in eventlet still tries to write to the connection after the client closed it. If this caused you real problems, other than just getting the error, let me know.

    • Multiple values of log_output. I think this one is a bug that I need to address. I'll look into it.

    • The KeyError when using gevent: I think this one is erroring because you are not using the websocket handler. See the documentation or the example application to see how this is done. Note that this is not a fatal error, it just means that the application will run in long-polling mode. I'll look into providing a more explicit error message to replace the key error.

    • The unexpected argument 'async_mode'. This one I need to investigate. Sounds like a bug I need to fix, but I have never seen this in my own tests, so I'm not sure. I may need to ask you for help reproducing the problem.

    Thank you, you have provided very useful information. My top task right now is the console output in debug mode, once I address that I will release an update.

  • #21 MyViz said

    Hello,

    Great job. I tested the library on a pcDuino board. It runs Python 2.7.3. My Python program with Flask-socketio sends telemetry data from a robot every 0.1 s to a browser using Javascript socket.io (and Flot for real-time plot) and it works very well.
    Thanks !

  • #22 John Duprey said

    This seems to work fine for me with python 3.4.3! Please promote it so we don't have to remember pip install flask-socketio==1.0a1 ! :)

  • #23 Miguel Grinberg said

    @John: I will make an official release soon. I have a couple small improvements I want to make and then it'll be out. Thanks for the feedback.

  • #24 Felix said

    Hi Miguel, I am also getting Max' KeyError even though I have gevent-websocket installed. Can you be a bit more specific about your answer, i.e. how to properly use the websocket handler with gevent?

  • #25 Miguel Grinberg said

    @Felix: I assume you are using gunicorn, correct? See the documentation for the command line you need to use to start the server: https://flask-socketio.readthedocs.org/en/latest/#deployment

Leave a Comment