how to get webpage window data with python?

Question

I am looking for a way to access a website's window varibles through python how can I do this?

For example, I want to get the data from window.performance. How can I do that with python? Is it possible to use the requests library?

I have attached the image:

Your problem has nothing to do with Python or the Requests library. This question is about how to send a HTTP request that will cause the data you want to be returned from the site, and then parsing that data out of the response. Just about any HTTP client library in any language is probably going to facilitate that. I'm not a web guy, but I think you're talking about JavaScript functions that are referenced from Javascript running inside a web page. Making a HTTP request isn't going to ever give you access to this for a general web site. Rather, the site would have to publish that info. — CryptoFool
– CryptoFool, Commented Jun 12, 2019 at 2:25
...unless you want to use Python to script the running of a web browser, cause it to load a page, and then query its API to have it look into the guts of the page and give you this information. My guess that you're not looking to go through all this work — CryptoFool
– CryptoFool, Commented Jun 12, 2019 at 2:30

alxgmpr · Accepted Answer · 2019-06-12 21:33:51Z

1

Python requests doesn't have access to running JavaScript on the page, which is what you'll need to get metrics like that. Requests will only gather the raw HTML data (what you see when you view the page source), and it won't be able to gather the JavaScript window object.

To do such a thing like this, you'll need a headless browser, which has access to inline JavaScript on the website. Something like this might interest you.

Regardless, you should look into using a headless browser which would have access to the live rendered site. Something like Selenium, Nightmare, Puppeteer, etc. would be of good use in this instance.

answered Jun 12, 2019 at 21:33

alxgmpr

14410 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

sdfs · Accepted Answer · 2020-08-26 14:10:18Z

0

the case is retrieving data which web app stores in the window js object or global scope in browser. The workflow is programming or instructing a headless browser with corresponding API in your language. #programming##data retrieve##access the window object in browser#

edited Aug 26, 2020 at 14:10

answered Aug 26, 2020 at 14:04

sdfs

11 bronze badge

Collectives™ on Stack Overflow

how to get webpage window data with python?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related