Python scraping Websocket datas

Question

I'm trying to scrape websocket datas(frames) from a website using sockjs, in Python but I don't really know how to do that.

URL: ws://example.io/sockjs/wkzeza/websocket

In the web debugger I can see this response headers:

Date: Sun, 27 Aug 2017 09:42:15 GMT
Connection: upgrade
Set-Cookie: oWG+Kel2MBo0v9FQK81NvuvBZcUwChaMvG2bsv1Ofs9Q8hHN+PlTn6PolO/8MgFXh2ygqC7A8WsJ7cgZwvpwvsbSp0VCpRHqmYMhGGxr; Expires=Sun, 03 Sep 2017 09:42:15 GMT; Path=/
Upgrade: websocket
Sec-WebSocket-Accept: HA0gkvrFCF7qjVYIDvSBa5sJKkg=
Sec-WebSocket-Extensions: permessage-deflate
Server: nginx
CF-RAY: 394e146d34a12f65-MAD

Normally with only the response header I can retrieve the datas from the frames, right?

I've tried with this code but I can read the content:

from websocket import create_connection
import json

headers = json.dumps({'Date': 'Sun, 27 Aug 2017 09:42:15 GMT',
'Connection': 'upgrade',
'Set-Cookie': 'oWG+Kel2MBo0v9FQK81NvuvBZcUwChaMvG2bsv1Ofs9Q8hHN+PlTn6PolO/8MgFXh2ygqC7A8WsJ7cgZwvpwvsbSp0VCpRHqmYMhGGxr; Expires=Sun, 03 Sep 2017 09:42:15 GMT; Path=/',
'Upgrade': 'websocket',
'Sec-WebSocket-Accept': 'HA0gkvrFCF7qjVYIDvSBa5sJKkg=',
'Sec-WebSocket-Extensions': 'permessage-deflate',
'Server': 'nginx',
'CF-RAY': '394e146d34a12f65-MAD'})

ws = create_connection('ws://example.io/sockjs/wkzeza/websocket', header=headers)
response = ws.recv_data_frame()
print(response)

>> [1, <websocket._abnf.ABNF at 0x7efe29aa0da0>]

Thanks for your help.

Well, finally I've found. The reason was... my lack of knowlegde about websocket and also I've found a really good repo with an example an explanation (github.com/oliver006/sockpuppet). With few changes it works perfectly for my case. — Matt
– Matt, Commented Aug 28, 2017 at 10:47

Punnerud · Accepted Answer · 2019-01-17 22:00:39Z

7

Check the traffic in Chrome (or other browser) to see how you should negotiate to beginning the flow of data. When the negotiation is OK you can do something like:

while True:
    ws.recv()

Here is an example for up/down WebSocket-traffic in Chrome.

Just copy the message up and use it in ws.send(). Example:

ws.send('''{"H":"publicmaphub","M":"getData","A":[],"I":1}''')

The example is from this live view of buses in Norway/Stavanger: https://www.kolumbus.no/ruter/kart/sanntidskart-internt/?c=58.974238,5.691347,14&lf=all&vt=bus,ferry
(On that page you also need to first get a token through HTTPS, connect with WebSocket and do another HTTPS to start the traffic. After this you can do the ws.recv() and ws.send() combos to start getting data.)

answered Jan 17, 2019 at 22:00

Punnerud

8,2312 gold badges58 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python scraping Websocket datas

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related