How to handle cgi form with Python requests

Question

I'm trying to use the requests module in Python to handle a cgi and can't work out what I've done wrong.

I've tried to use Google Dev Tools in Chrome to provide the right params and data but I've not quite fixed it.

The site I'm trying to get data from is: http://staffordshirebmd.org.uk/cgi/birthind.cgi

Here's my code

import requests 

headers = {"Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
        "Accept-Encoding":"gzip,deflate,sdch",
        "Accept-Language":"en-US,en;q=0.8",
        "Cache-Control":"no-cache",
        "Connection":"keep-alive",
        "Content-Length":"124",
        "Content-Type":"application/x-www-form-urlencoded",
        "DNT":"1",
        "Host":"staffordshirebmd.org.uk",
        "Origin":"http://staffordshirebmd.org.uk",
        "Pragma":"no-cache",
        "Referer":"http://staffordshirebmd.org.uk/cgi/birthind.cgi?county=staffordshire",
        "User-Agent":"Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25"}

payload = {"county":"staffordshire",
          "lang": "",
          "year_date":"1837",
          "search_region":"All",
          "sort_by":"alpha",
          "csv_or_list":"screen",
          "letter":"A",
          "submit":"Display Indexes"}

f = requests.put(path, data=payload, headers=headers)

f.text

This provides the response:

u'<html>\n<body>\n<div>\n<p>\nThe Bookmark you have used to reach this page is not valid.\n</p>\n<p>\nPlease click <a href="http://staffordshirebmd.org.uk/">here</a> to return to the main page and reset your\nbookmark to that page.\n</p>\n</div>\n</body>\n</html>\n\n'

What am I doing wrong?

Don't set the content-length and content type; those are handled for you just fine. — Martijn Pieters
– Martijn Pieters, Commented Apr 23, 2014 at 15:35
Thank you - I clearly wasn't concentrating. Put post and it works a treat. — elksie5000
– elksie5000, Commented Apr 23, 2014 at 15:37

Martijn Pieters · Accepted Answer · 2014-04-23 15:40:37Z

2

The URL you used in your Referrer header has a form that uses POST, not PUT.

You rarely should hardcode the Content-Length header; do leave that up to requests to calculate for you and set. Different browsers can easily use subtly different content lengths, a script that only works with a fixed Content-Length would not last long.

Removing the Content-Length header and changing .put() to .post() gives me results, in any case.

answered Apr 23, 2014 at 15:40

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

elksie5000 Over a year ago

Thanks for that. I appreciate the tip about Content-Length as well. Wasn't quite sure what to put in to get the results. :)

Collectives™ on Stack Overflow

How to handle cgi form with Python requests

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related