Post forms using requests on .net website (python)

Question

import requests

headers ={
"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Encoding":"gzip, deflate",
"Accept-Language":"en-US,en;q=0.5",
"Connection":"keep-alive",
"Host":"mcfbd.com",
"Referer":"https://mcfbd.com/mcf/FrmView_PropertyTaxStatus.aspx",
"User-Agent":"Mozilla/5.0(Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0"}

a = requests.session()
soup = BeautifulSoup(a.get("https://mcfbd.com/mcf/FrmView_PropertyTaxStatus.aspx").content)

payload = {"ctl00$ContentPlaceHolder1$txtSearchHouse":"",
"ctl00$ContentPlaceHolder1$txtSearchSector":"",
"ctl00$ContentPlaceHolder1$txtPropertyID":"",
"ctl00$ContentPlaceHolder1$txtownername":"",
"ctl00$ContentPlaceHolder1$ddlZone":"1",
"ctl00$ContentPlaceHolder1$ddlSector":"2",
"ctl00$ContentPlaceHolder1$ddlBlock":"2",
"ctl00$ContentPlaceHolder1$btnFind":"Search",
"__VIEWSTATE":soup.find('input',{'id':'__VIEWSTATE'})["value"],
"__VIEWSTATEGENERATOR":"14039419",
"__EVENTVALIDATION":soup.find("input",{"name":"__EVENTVALIDATION"})["value"],
"__SCROLLPOSITIONX":"0",
"__SCROLLPOSITIONY":"0"}

b = a.post("https://mcfbd.com/mcf/FrmView_PropertyTaxStatus.aspx",headers = headers,data = payload).text
print(b)

above is my code for this website.

https://mcfbd.com/mcf/FrmView_PropertyTaxStatus.aspx

I checked firebug out and these are the values of the form data. however doing this:

b = requests.post("https://mcfbd.com/mcf/FrmView_PropertyTaxStatus.aspx",headers = headers,data = payload).text
print(b)

throws this error:

[ArgumentException]: Invalid postback or callback argument

is my understanding of submitting forms via request correct?

1.open firebug

2.submit form

3.go to the NET tab

4.on the NET tab choose the post tab

5.copy form data like the code above

I've always wanted to know how to do this. I could use selenium but I thought I'd try something new and use requests

Vikas Ojha · Accepted Answer · 2015-09-17 21:02:43Z

2

The error you are receiving is correct because the fields like _VIEWSTATE (and others as well) are not static or hardcoded. The proper way to do this is as follows:

Create a Requests Session object. Also, it is advisable to update it with headers containing USER-AGENT string -

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36",}`
s = requests.session()

Navigate to the specified url -

r = s.get(url)

Use BeautifulSoup4 to parse the html returned -

from bs4 import BeautifulSoup
soup = BeautifulSoup(r.content, 'html5lib')

Populate formdata with the hardcoded values and dynamic values -

formdata = {
   '__VIEWSTATE': soup.find('input', attrs={'name': '__VIEWSTATE'})['value'],
   'field1': 'value1'
}

Then send the POST request using the session object itself -

s.post(url, data=formdata)

edited Sep 17, 2015 at 21:02

answered Sep 17, 2015 at 20:40

Vikas Ojha

7,0006 gold badges24 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Vikas Ojha Over a year ago

We are using Session object to maintain session (i.e. cookies)

Zion Over a year ago

does the order matter? I mean the order of the parameters? no right?

Vikas Ojha Over a year ago

No the order doesnt matter at all.

Zion Over a year ago

one more thing I get this error TypeError: 'module' object is not callable doing this a = requests.sessions()

Vikas Ojha Over a year ago

It is requests.session() and not sessions ... notice an unwanted s at the end.

|

Collectives™ on Stack Overflow

Post forms using requests on .net website (python)

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related