Scrape socket.io data from website using python

Question

I am trying to scrape latest bid and ask prices from this website "http://btc-exchange.com/" I can see that the prices are being provided by this socket.io

wss://pusher.mistertango.com/socket.io/?EIO=3&transport=websocket&sid=XXX

The sessionID is being generated from this call

https://pusher.mistertango.com/socket.io/?EIO=3&transport=polling&t=1517079662330-10

This is the code which I am currently using

import requests
from websocket import create_connection
import json

SID_url = "https://pusher.mistertango.com/socket.io/?EIO=3&transport=polling"
SID_req = requests.get(SID_url, headers={'User-Agent': 'Mozilla/5.0'}).text
SID = SID_req[SID_req.index("sid")+6:SID_req.index(",")-1]
print(SID_req)
print(SID)

ws = create_connection("wss://pusher.mistertango.com/socket.io/?EIO=3&transport=websocket&sid="+SID)
ws.send('2probe')
print(ws.recv())
ws.send('5')
print(ws.recv())
ws.send('42["subscribe",{"chan":"market-e559906eda4362f58bcaab40a4bfb5b4"}]')
while True:
    result = ws.recv()
    print(result)
ws.close()

This is the output of the code

    ÿ0{"sid":"mURV8OnaNqax_AmvAAF2","upgrades":["websocket"],"pingInterval":25000,"pingTimeout":60000}
mURV8OnaNqax_AmvAAF2
3probe
40

The messages which I have sent are based on the messages I saw in chrome-dev tool. Websocket msgs

My connection stops receiving any message after '40'. What am I doing wrong?

Did you find a solution to your problem eventually?

Vincent K
– Vincent K

2018-03-07 11:14:35 +00:00
Commented Mar 7, 2018 at 11:14 — Vincent K
– Vincent K, Commented Mar 7, 2018 at 11:14

Tadas Kvedaras · Accepted Answer · 2018-07-05 12:21:17Z

3

There is code of what you want. Tested and works

#!/usr/bin/env python
from socketIO_client import SocketIO

def on_connect():
    print('connect')

def on_disconnect():
    print('disconnect')

def on_reconnect():
    print('reconnect')

def on_aaa_response(*args):
    print('on_aaa_response', args)

def on_bbb_response(*args):
    print('on_bbb_response', args)

with SocketIO('https://pusher.mistertango.com') as socketIO:
    socketIO.on('connect', on_connect)
    socketIO.on('disconnect', on_disconnect)
    socketIO.on('reconnect', on_reconnect)
    socketIO.on('market-orderbook', on_aaa_response)
    socketIO.emit('subscribe', {'chan': 'market-orderbook'}, on_bbb_response)
    socketIO.wait_for_callbacks(seconds=10)

edited Jul 5, 2018 at 12:21

answered Jul 5, 2018 at 9:39

Tadas Kvedaras

5573 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Ethan Over a year ago

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review

Tadas Kvedaras Over a year ago

Changed my answer

ravimalik20 · Accepted Answer · 2018-01-27 23:12:54Z

0

You can use asyncio and websocket to solve your problem. You can use the following command to install websocket

pip install websocket

After that use the following code snippet as an example to communicate with the web server:

import asyncio
import websockets

async def responseFunc(uri):
    async with websockets.connect(uri) as websocket:
        await websocket.send("Hello world!")

url = "wss://pusher.mistertango.com/socket.io/?EIO=3&transport=websocket&sid=XXX"

asyncio.get_event_loop().run_until_complete(
    responseFunc(url))

Ref: https://pypi.python.org/pypi/websockets

answered Jan 27, 2018 at 23:12

ravimalik20

779 bronze badges

1 Comment

arjun sachdeva Over a year ago

Thanks for the answer but I am still not receiving any response using this. I am able to establish connection but not receiving response.

Collectives™ on Stack Overflow

Scrape socket.io data from website using python

2 Answers 2

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related