Sending HTTP Post Requests work in Fiddler but not in Python

Question

I am sending several post requests in Fiddler2 to check my site to make sure it is working properly. However, when I automate in Python to simulate this over several hours (I really don't want to spend 7 hours hitting space!).

This works in fiddler. I can create the account and perform related API commands. However in Python, nothing happens with this code:

def main():
    import socket
    from time import sleep
    x = raw_input("Points: ")
    x = int(x)
    x = int(x/150)
    for y in range(x):
        new = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        new.connect(('example.com', 80))
        mydata ="""POST http://www.example.com/api/site/register/ HTTP/1.1
Host: www.example.com
Connection: keep-alive
Content-Length: 191
X-NewRelic-ID: UAIFVlNXGwEFV1hXAwY=
Origin: http://www.example.com
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.154 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Accept: application/json, text/javascript, */*; q=0.01
X-Requested-With: XMLHttpRequest
X-CSRFToken: CEC9EzYaQOGBdO9HGPVVt3Fg66SVWVXg
DNT: 1
Referer: http://www.example.com/signup
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en;q=0.8
Cookie: sessionid=sessionid; sb-closed=true; arp_scroll_position=600; csrftoken=2u92jo23g929gj2; __utma=912.1.1.2.5.; __utmb=9139i91; __utmc=2019199; __utmz=260270731.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)

username=user&password=password&moredata=here """
        new.send(mydata.encode('hex'))
        print "Sent", y, "of", x
        sleep(1)
    print "Sent all!"
    print "restarting"
    main()
main()

I know I could use While True, but I intend to add more functions later to test more sites.

Why does this program not do anything to the API, when Fiddler2 can? I know it is my program, as I can send the exact same packet in fiddler (obviously pointing to the right place) and it works.

PS - If anyone does fix this, as its probably something really obvious, please can you only use modules that are bundled with Python. I cannot install modules from other places. Thanks!

I can't see how mydata is exactly the same as the HTTP request you've made above. You don't send any headers (including POST http://www.example.com/api/site/register/ HTTP/1.1). Perhaps you should write exactly what you put in mydata. Plus why do you .encode('hex') data? HTTP is a raw string. Finally: you should use some high level library (instead of low level sockets) like this one: docs.python.org/2/library/httplib.html — freakish
– freakish, Commented Mar 17, 2014 at 22:48
Mydata is the packet above, just that it breaks formatting for some reason if I include it in the code, so I used a separate chunk. Also, because the site I used said to encode it as HEX. Is this the problem? — George
– George, Commented Mar 17, 2014 at 22:53
@freakish. I saved the packet above as a multi-line variable. Fixed it! :D — George
– George, Commented Mar 17, 2014 at 22:53
Good. Much better. Now what you don't know is how HTTP requests are made. I'll answer your question soon. — freakish
– freakish, Commented Mar 17, 2014 at 22:55
Thanks :D Would whoever has downvoted please explain why. I am open to criticism, as long as its constuctive, so I can write better questions next time. — George
– George, Commented Mar 18, 2014 at 16:08

George · Accepted Answer · 2014-03-18 16:23:54Z

HTTP requests are not as easy as you think they are. First of all this is wrong:

"""POST http://www.example.com/api/site/register/ HTTP/1.1
Host: www.example.com
Connection: keep-alive
...
"""

Each line in HTTP request has to end with CRLF (in Python with \r\n), i.e. it should be:

"""POST http://www.example.com/api/site/register/ HTTP/1.1\r
Host: www.example.com\r
Connection: keep-alive\r
...
"""

Note: LF = line feed = \n is there implicitly. Also you didn't see CR in your fiddler, because it's a white-space. But it has to be there (simple copy-paste won't work).

Also HTTP specifies that after headers there has to be CRLF as well. I.e. your entire request should be:

    mydata = """POST http://www.example.com/api/site/register/ HTTP/1.1\r
Host: www.example.com\r
Connection: keep-alive\r
Content-Length: 191\r
X-NewRelic-ID: UAIFVlNXGwEFV1hXAwY=\r
Origin: http://www.example.com\r
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.154 Safari/537.36\r
Content-Type: application/x-www-form-urlencoded; charset=UTF-8\r
Accept: application/json, text/javascript, */*; q=0.01\r
X-Requested-With: XMLHttpRequest\r
X-CSRFToken: CEC9EzYaQOGBdO9HGPVVt3Fg66SVWVXg\r
DNT: 1\r
Referer: http://www.example.com/signup\r
Accept-Encoding: gzip,deflate,sdch\r
Accept-Language: en-GB,en;q=0.8\r
Cookie: sessionid=sessionid; sb-closed=true; arp_scroll_position=600; csrftoken=2u92jo23g929gj2; __utma=912.1.1.2.5.; __utmb=9139i91; __utmc=2019199; __utmz=260270731.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)\r
\r
username=user&password=password&moredata=here"""

Warning: it should be exactly as I've written. There can't be any spaces in front of each line, i.e. this:

    mydata = """POST http://www.example.com/api/site/register/ HTTP/1.1\r
    Host: www.example.com\r
    Connection: keep-alive\r
    ...
"""

is wrong.

Side note: you can move mydata to the top, outside of the loop. Unimportant optimization but makes your code cleaner.

Now you've said that the site you are using wants you to hex-encode HTTP request? It's hard for me to believe that (HTTP is a raw string by definition). Don't do that (and ask them to specify what exactly this hex-encoding means). Possibly they've meant that the URL should be hex-encoded (since it is the only hex-encoding that is actually used in HTTP)? In your case there is nothing to encode so don't worry about it. Just remove the .encode('hex') line.

Also Content-Length header is messed up. It should be the actual length of the content. So if for example body is username=user&password=password&moredata=here then it should be Content-Length: 45.

Next thing is that the server might not allow you making multiple requests without getting a response. You should use new.recv(b) where b is a number of bytes you want to read. But how many you should read? Well this might be problematic and that's where Content-Length header comes in. First you have to read headers (i.e. read until you read \r\n\r\n which means the end of headers) and then you have to read the body (based on Content-Length header). As you can see things are becoming messy (see: final section of my answer).

There are probably more issues with your code. For example X-CSRFToken suggests that the site uses CSRF prevention mechanism. In that case your request might not work at all (you have to get the value of X-CSRFToken header from the server).

Finally: don't use sockets directly. Httplib (http://docs.python.org/2/library/httplib.html) is a great (and built-in) library for making HTTP requests which will deal with all the funky and tricky HTTP stuff for you. Your code for example may look like this:

import httplib

headers = {
    "Host": "www.example.com",
    "X-NewRelic-ID": "UAIFVlNXGwEFV1hXAwY=",
    "Origin": "http://www.example.com",
    "Connection": "keep-alive",
    "User-Agent": "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.154 Safari/537.36",
    "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
    "Accept": "application/json, text/javascript, */*; q=0.01",
    "X-Requested-With": "XMLHttpRequest",
    "X-CSRFToken": "CEC9EzYaQOGBdO9HGPVVt3Fg66SVWVXg",
    "DNT": "1",
    "Referer": "http://www.example.com/signup",
    "Accept-Encoding": "gzip,deflate,sdch",
    "Accept-Language": "en-GB,en;q=0.8",
    "Cookie": "sessionid=sessionid; sb-closed=true; arp_scroll_position=600; csrftoken=2u92jo23g929gj2; __utma=912.1.1.2.5.; __utmb=9139i91; __utmc=2019199; __utmz=260270731.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)"
}

body = "username=user&password=password&moredata=here"

conn = httplib.HTTPConnection("example.com")
conn.request("POST", "http://www.example.com/api/site/register/", body, headers)
res = conn.getresponse()

Note that you don't need to specify Content-Length header.

Thanks for the answer. One thing that I have no idea about is how Memrise got in there! My mistake - edited the question slightly. Thanks for the great and well explained answer. +1 to you :)
I don't intend to cause any offence, but to save time for you I changed the host manually. If this causes any problems then of course I will revert it :)
I see. I need to save it in a dictionary! It all makes sense now - thanks :D
use an even higher level library,l like urllib2 or requests.

Collectives™ on Stack Overflow

Sending HTTP Post Requests work in Fiddler but not in Python

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related