How to scrape a website which requires login using python

Question

i want to webscrpe this website and i need to login to get the account datail (https://online.cspension.com.hk/hkpensionweb/#/login?source=corporateHK&vgnLocale=en_CA), i don't want to use selenium

how to login and get the account datail

Flow · Accepted Answer · 2022-11-02 05:08:39Z

3

You can use a requests.Session() instance to make a post request to the login url with your login details as a payload. Making requests from a session instance is essentially the same as using requests normally, it simply adds persistence, allowing you to store and use cookies etc.

import requests

# Fill in your details here to be posted to the login form.
payload = {
    'inUserName': 'username',
    'inUserPass': 'password'
}

# Use 'with' to ensure the session context is closed after use.
with requests.Session() as s:
    p = s.post('LOGIN_URL', data=payload)
    # print the html returned or something more intelligent to see if it's a successful login page.
    print p.text

    # An authorised request.
    r = s.get('A protected web page url')
    print r.text
        # etc...

answered Nov 2, 2022 at 5:08

Flow

1,0701 gold badge13 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to scrape a website which requires login using python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related