0

I am using a websocket server coded in Python 3.5. This is needed to provide my website realtime information.

The "server"-library I use is called tornado (version 4.5b2) which handles the websocket connection and http requests. For backwards compatibility (if a browser does not support websocket yet) I use the python library sockjs-tornado (version 1.0.3) which also adds some additional features to the websocket connection.

Now my problem is that I get a lot of tornado ssl errors in the console if I start the server. It does not crash, but if about 80 clients are connected to the websocket 3 errors occur quite often. I use an encrypted connection and it works, but some clients are kicked out of the websocket connection if one of theese errors occur.

The Python script uses multiple threads and the server is protected by cloudflare, which is some kind of middleman and redirects every request to the server or user.

These are the 3 errors:

​
ERROR:tornado.general:Uncaught exception, closing connection.
Traceback (most recent call last):
  File "/home/website/python/tornado/iostream.py", line 523, in _handle_events
    self._handle_write()
  File "/home/website/python/tornado/iostream.py", line 1398, in _handle_write
    super(SSLIOStream, self)._handle_write()
  File "/home/website/python/tornado/iostream.py", line 847, in _handle_write
    assert self._write_buffer_size >= 0
AssertionError
ERROR:tornado.application:Exception in callback None
Traceback (most recent call last):
  File "/home/website/python/tornado/ioloop.py", line 888, in start
    handler_func(fd_obj, events)
  File "/home/website/python/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/website/python/tornado/iostream.py", line 523, in _handle_events
    self._handle_write()
  File "/home/website/python/tornado/iostream.py", line 1398, in _handle_write
    super(SSLIOStream, self)._handle_write()
  File "/home/website/python/tornado/iostream.py", line 847, in _handle_write
    assert self._write_buffer_size >= 0
AssertionError

This error does not appear as often as the first one.

ERROR:tornado.application:Uncaught exception GET /sockjs/398/ssputw1s/websocket
HTTPServerRequest(protocol='https', host='****.com:8443', method='GET', uri='/sockjs/398/ssputw1s/websocket', version='HTTP/1.1', remote_ip='****', headers={'Accept-Encoding': 'gzip', 'Cf-Visitor': '{"scheme":"https"}', 'Sec-Websocket-Key': '****', 'Host': '****.com:8443', 'X-Forwarded-Proto': 'https', 'Cache-Control': 'no-cache', 'Sec-Websocket-Version': '13', 'Cf-Ipcountry': 'FR', 'X-Forwarded-For': '****', 'Sec-Websocket-Extensions': 'x-webkit-deflate-frame', 'Origin': 'https://****.com', 'Cf-Connecting-Ip': '****', 'Cookie': '****', 'Upgrade': 'websocket', 'Cf-Ray': '34edb1a17c456908-CDG', 'User-Agent': '****', 'Connection': 'Upgrade', 'Pragma': 'no-cache'})
Traceback (most recent call last):
  File "/home/website/python/tornado/web.py", line 1464, in _stack_context_handle_exception
    raise_exc_info((type, value, traceback))
  File "<string>", line 4, in raise_exc_info
  File "/home/website/python/tornado/stack_context.py", line 316, in wrapped
    ret = fn(*args, **kwargs)
  File "/home/website/python/tornado/websocket.py", line 865, in _on_masked_frame_data
    self._on_frame_data(_websocket_mask(self._frame_mask, data))
  File "/home/website/python/tornado/websocket.py", line 910, in _on_frame_data
    self._receive_frame()
  File "/home/website/python/tornado/websocket.py", line 784, in _receive_frame
    self.stream.read_bytes(2, self._on_frame_start)
  File "/home/website/python/tornado/iostream.py", line 324, in read_bytes
    self._try_inline_read()
  File "/home/website/python/tornado/iostream.py", line 711, in _try_inline_read
    pos = self._read_to_buffer_loop()
  File "/home/website/python/tornado/iostream.py", line 625, in _read_to_buffer_loop
    if self._read_to_buffer() == 0:
  File "/home/website/python/tornado/iostream.py", line 738, in _read_to_buffer
    chunk = self.read_from_fd()
  File "/home/website/python/tornado/iostream.py", line 1487, in read_from_fd
    chunk = self.socket.read(self.read_chunk_size)
  File "/usr/lib64/python3.5/ssl.py", line 799, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib64/python3.5/ssl.py", line 585, in read
    v = self._sslobj.read(len)
ssl.SSLWantWriteError: The operation did not complete (write) (_ssl.c:2090)

And sometimes this error appears:

ERROR:tornado.general:Uncaught exception, closing connection.
Traceback (most recent call last):
  File "/home/website/python/tornado/iostream.py", line 523, in _handle_events
    self._handle_write()
  File "/home/website/python/tornado/iostream.py", line 1398, in _handle_write
    super(SSLIOStream, self)._handle_write()
  File "/home/website/python/tornado/iostream.py", line 872, in _handle_write
    del self._write_buffer[:self._write_buffer_pos]
BufferError: Existing exports of data: object cannot be re-sized
ERROR:tornado.application:Exception in callback None
Traceback (most recent call last):
  File "/home/website/python/tornado/ioloop.py", line 888, in start
    handler_func(fd_obj, events)
  File "/home/website/python/tornado/stack_context.py", line 277, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/website/python/tornado/iostream.py", line 523, in _handle_events
    self._handle_write()
  File "/home/website/python/tornado/iostream.py", line 1398, in _handle_write
    super(SSLIOStream, self)._handle_write()
  File "/home/website/python/tornado/iostream.py", line 872, in _handle_write
    del self._write_buffer[:self._write_buffer_pos]
BufferError: Existing exports of data: object cannot be re-sized

2 Answers 2

1

This looks like a bug in Tornado 4.5b2 which is being discussed in this issue. Look for it to be fixed in 4.5b3 or 4.5 final, and in the meantime you can go back to version 4.4.

Sign up to request clarification or add additional context in comments.

3 Comments

Actually, since you say you're using multiple threads, this looks like it could be a result of using threads incorrectly. You must only call methods of WebSocketHandler from the IOLoop thread; you may not write to the socket from any other thread.
Is there an example of multiple threads wanting to send a message on the websocket?
I'm not aware of any complete example of this (remember that the intended use of Tornado is to not use multiple threads). But if you want to use threads, any number of threads may use IOLoop.add_callback to schedule calls to WebSocketHandler.write_message to be run on the main thread.
0

Looks like a bug in the library - namely, race conditions that cause multiple entities to use the same write buffer at the same time.

  • 1st error - _write_buffer_size underflows - another entity reads from it when not appropriate
  • 3rd error - buffer in use when it shouldn't be

To catch it, you need to trace all operations that use _write_buffers and change _write_buffer_sizes and where they're coming from.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.