Python selenium web scraping - freeze page

Question

I am scraping a few pages with selenium, and I do not use other frameworks (like scrapy, etc..) because of a lot of ajax action. My problem is that the content is refreshing automatically nearly every second (like for example financial data) but I want to scrape all the elements in a static state. I searched alot in the internet and especially here on stackoverflow. WHat is the easiest way to freeze the website with selenium? I even tried switching off the wireless adapter but this was a problem... This is the only command in the selenium docs that I found:

driver.set_network_conditions(offline=True, latency=5, throughput=500 * 1024)

I tested this code and when i run the script it doesn't have any effect. The website is still "auto refreshing"...

for example this one: gate hub.net/markets/XRP/USD+rhub8VRN55s94qWKDv6jmDy1pUykJzF3wq (there is no API for this site) — robrados
– robrados, Commented Feb 6, 2019 at 20:10
Is this that you need? api.gatehub.net/rippledata/v2/exchanges/… You can increase the limit parameter if needed (tested max 400). — Pedro Lobito
– Pedro Lobito, Commented Feb 6, 2019 at 21:23

Pedro Lobito · Accepted Answer · 2019-02-07 11:42:27Z

1

"for example this one: https://gatehub.net/markets/XRP/USD+rhub8VRN55s94qWKDv6jmDy1pUykJzF3wq (there is no API for this site)"

In fact, an api exists, but it isn't fully public.

To get the values of the chart as a json object, you'll need to construct a customized URL, something like:

https://api.gatehub.net/rippledata/v2/exchanges/USD+rhub8VRN55s94qWKDv6jmDy1pUykJzF3wq/XRP?descending=true&end=2019-02-06T21:20:00.000Z&limit=400&reduce=false&result=tesSUCCESS&start=2009-02-06T21:20:00.000Z

Output:

{"result":"success","count":400,"marker":"USD|rhub8VRN55s94qWKDv6jmDy1pUykJzF3wq|XRP||20190206014150|000044926668|00006|00003","exchanges":[{"base_amount":"0.12180204","counter_amount":"0.42056","node_index":6,"rate":"3.4528157","tx_index":18,"autobridged_currency":"ETH","autobridged_issuer":"rcA8X3TVMST1n3CJeAdGk1RdRCHii7N2h","buyer":"rGmGFAEx1hYEJuSAfrjEBdA48AXWJBMp1D","executed_time":"2019-02-06T21:14:00Z","ledger_index":44945715,"offer_sequence":39832,"provider":"rGmGFAEx1hYEJuSAfrjEBdA48AXWJBMp1D","seller":"rUmnnszuTRfhKnULCjcKzV7mJeazCF7Gik","taker":"rUmnnszuTRfhKnULCjcKzV7mJeazCF7Gik","tx_hash":"4E39DB1CB68B4635E773082042B47168094852ED4A11C93AED7F85A67F1F7EDD","tx_type":"OfferCreate","base_currency":"USD","base_issuer":"rhub8VRN55s94qWKDv6jmDy1pUykJzF3wq","counter_currency":"XRP"},{"base_amount":"322.8872040048709","counter_amount":"1109.37944","node_index":2,"rate":"3.4358111","tx_index":18,"autobridged_currency":"ETH","autobridged_issuer":"rcA8X3TVMST1n3CJeAdGk1RdRCHii7N2h","buyer":"rETx8GBiH6fxhTcfHM9fGeyShqxozyD3xe","executed_time":"2019-02-06T21:14:00Z","ledger_index":44945715,"offer_sequence":26918939,"provider":"rETx8GBiH6fxhTcfHM9fGeyShqxozyD3xe","seller":"rUmnnszuTRfhKnULCjcKzV7mJeazCF7Gik","taker":"rUmnnszuTRfhKnULCjcKzV7mJeazCF7Gik","tx_hash":"4E39DB1CB68B4635E773082042B47168094852ED4A11C93AED7F85A67F1F7EDD","tx_type":"OfferCreate","base_currency":"USD","base_issuer":"rhub8VRN55s94qWKDv6jmDy1pUykJzF3wq","counter_currency":"XRP"}

...

Notes:

You can change the limit parameter to display different number of records if needed (tested max 400)
Dates should also be automagically updated to get the latest values.

edited Feb 7, 2019 at 11:42

answered Feb 6, 2019 at 21:36

Pedro Lobito

99.8k36 gold badges274 silver badges278 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

robrados Over a year ago

thank you for your answer, this was helpful, although my question was how to stop the javascript / the autorefreshing the whole website/elements during a selenium driver session.

Pedro Lobito Over a year ago

Try injecting a js error via selenium execute_script: throw new Error();

Susie Queue · Accepted Answer · 2019-02-06 21:01:45Z

0

One solution might be to look into being able to set config preferences for whichever browser you are using for your driver. For example, if using Firefox you could set accessibility.blockautorefresh to False, and then just use driver.refresh() when you are ready.

https://lifehacker.com/disable-automatic-web-page-refreshing-5321420

PHPUnit + Selenium: How to set Firefox about:config options?

answered Feb 6, 2019 at 21:01

Susie Queue

361 silver badge3 bronze badges

Collectives™ on Stack Overflow

Python selenium web scraping - freeze page

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related