python web scraping post form data using requests not working

Question

I am trying to post input data into a form using a requests.session and it's returning a 500 status. I am expecting to see the search results retrieved.

I was able to get around a previous login issue with __RequestVerificationToken and cookies - thanks to the help of Bertrand Martel. The next step in my process is to get the Search page, which I was able to get successfully. Now failing when I try to post data into the date fields on the form, which make up the search criteria. Works when I manually complete the form and press submit. All seems very straightforward to me, but not sure why it won't work. Is it still a cookies issue? Any help would be appreciated.

Here is my code:

import requests
from bs4 import BeautifulSoup

EMAIL = '[email protected]'
PASSWORD = 'somepwd'
LOGIN_URL = 'https://www.idocmarket.com/Security/LogOn'
SEARCH_URL = 'https://www.idocmarket.com/RIOCO/Document/Search'


s = requests.Session()
s.get(LOGIN_URL)

result = s.post(LOGIN_URL, data = {
    "Login.Username": EMAIL,
    "Login.Password": PASSWORD
})
soup = BeautifulSoup(result.text, "html.parser")
# Report successful login
print("Login succeeded: ", result.ok)
print("Status code:", result.status_code)

result = s.get(SEARCH_URL)
auth_token  = soup.find("input", {'name': '__RequestVerificationToken'}).get('value')
print('auth token:', auth_token )
print("Get Search succeaeded: ", result.ok)
print("get Search Statusa code:", result.status_code)
result = s.post(SEARCH_URL, data = {
    "__RequestVerificationToken": auth_token,
    "StartRecordDate": "03/01/2019",
    "EndRecordDate": "03/31/2019",
    "StartDocNumber": "",
    "EndDocNumber": "",
    "Book": "",
    "Page": "",
    "Instrument": "",
    "InstrumentGroup": "",
    "PartyType": "Either",
    "PartyMatchType": "Contains",
    "PartyName": "",
    "Subdivision": "",
    "StartLot": "",
    "EndLot": "",
    "Block": "",
    "Section":"",
    "Township": "",
    "Range": "",
    "Legal": "",
    "CountyKey": "RIOCO"
})
print("post Dates succeeded: ", result.ok)
print("post Dates Status code:", result.status_code)
print(result.text)

Bertrand Martel · Accepted Answer · 2019-10-13 19:37:06Z

1

It seems that this time, the xsrf token is needed in the post along with all the existing parameters. A simple solution is to get all the input value & pass it to the request :

import requests
from bs4 import BeautifulSoup

LOGIN_URL = 'https://www.idocmarket.com/Security/LogOn'
SEARCH_URL = 'https://www.idocmarket.com/RIOCO/Document/Search'
EMAIL = '[email protected]'
PASSWORD = 'somepwd'

s = requests.Session()
s.get(LOGIN_URL)

r = s.post(LOGIN_URL, data = {
    "Login.Username": EMAIL,
    "Login.Password": PASSWORD
})

if (r.status_code == 200):
    r = s.get(SEARCH_URL)
    soup = BeautifulSoup(r.text, "html.parser")
    payload = {}
    for input_item in soup.select("input"):
        if input_item.has_attr('name'):
            payload[input_item["name"]] = input_item["value"]
    payload["StartRecordDate"] = '09/01/2019'
    payload["EndRecordDate"] = '09/30/2019'
    r = s.post(SEARCH_URL, data = payload)
    soup = BeautifulSoup(r.text, "html.parser")
    print(soup)
else:
    print("authentication failure")

Also using comprehension list for the payload you can write :

temp_pl = [
    (t['name'], t['value']) 
    for t in soup.select("input")
    if t.has_attr('name')
]
payload = dict(temp_pl)
payload["StartRecordDate"] = '09/01/2019'
payload["EndRecordDate"] = '09/30/2019'

edited Oct 13, 2019 at 19:37

answered Oct 13, 2019 at 19:29

Bertrand Martel

46k17 gold badges156 silver badges170 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

GlennG Over a year ago

Thank you Bertrand. I had just came to the same conclusion and updated my code . Although, your code solution regarding supplying the form input fields is much more elegant . Thanks again for the help.

Collectives™ on Stack Overflow

python web scraping post form data using requests not working

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related