upload binary data or file from memory using python http post

import io         # Part of core Python
import requests   # Install via: 'pip install requests'

# Get the data in bytes. I got it via:
#   with open("smile.gif", "rb") as fp: data = fp.read()
data = b"GIF89a\x12\x00\x12\x00\x80\x01\x00\x00\x00\x00\xff\x00\x00!\xf9\x04\x01\n\x00\x01\x00,\x00\x00\x00\x00\x12\x00\x12\x00\x00\x024\x84\x8f\x10\xcba\x8b\xd8ko6\xa8\xa0\xb3Wo\xde9X\x18*\x15x\x99\xd9'\xad\x1b\xe5r(9\xd2\x9d\xe9\xdd\xde\xea\xe6<,\xa3\xd8r\xb5[\xaf\x05\x1b~\x8e\x81\x02\x00;"

# Hookbin is similar to requestbin - just a neat little service
# which helps you to understand which queries are sent
url = 'https://hookb.in/je2rEl733Yt9dlMMdodB'

in_memory_file = io.BytesIO(data)
response = requests.post(url, files=(("smile.gif", in_memory_file),))

This (small) library will take a file descriptor, and will do the HTTP POST operation: https://github.com/seisen/urllib2_file/

You can pass it a StringIO object (containing your in-memory data) as the file descriptor.

Find something that cas work with a file handle. Then simply pass a StringIO object instead of a real file descriptor.

I met similar issue today, after tried both and pycurl and multipart/form-data, I decide to read python httplib/urllib2 source code to find out, I did get one comparably good solution:

1. set Content-Length header(of the file) before doing post
2. pass a opened file when doing post

Here is the code:

import urllib2, os
image_path = "png\\01.png"
url = 'http://xx.oo.com/webserviceapi/postfile/'
length = os.path.getsize(image_path)
png_data = open(image_path, "rb")
request = urllib2.Request(url, data=png_data)
request.add_header('Cache-Control', 'no-cache')
request.add_header('Content-Length', '%d' % length)
request.add_header('Content-Type', 'image/png')
res = urllib2.urlopen(request).read().strip()
return res

see my blog post: http://www.2maomao.com/blog/python-http-post-a-binary-file-using-urllib2/

Following python code works reliable on 2.6.x. The input data is of type str. Note that the server that receives the data has to loop to read all the data as the large data sizes will be chunked. Also attached java code snippet to read the chunked data.

def post(self, url, data):
    self.curl = pycurl.Curl()
    self.response_headers = StringIO.StringIO()
    self.response_body = io.BytesIO()
    self.curl.setopt(pycurl.WRITEFUNCTION, self.response_body.write)
    self.curl.setopt(pycurl.HEADERFUNCTION, self.response_headers.write)
    self.curl.setopt(pycurl.FOLLOWLOCATION, 1)
    self.curl.setopt(pycurl.MAXREDIRS, 5)
    self.curl.setopt(pycurl.TIMEOUT, 60)
    self.curl.setopt(pycurl.ENCODING,"deflate, gzip")
    self.curl.setopt(pycurl.URL, url)
    self.curl.setopt(pycurl.VERBOSE, 1)
    self.curl.setopt(pycurl.POST,1)
    self.curl.setopt(pycurl.POSTFIELDS,data)
    self.curl.setopt(pycurl.HTTPHEADER, [ "Content-Type: octate-stream" ])
    self.curl.setopt(pycurl.POSTFIELDSIZE, len(data))
    self.curl.perform()
    return url, self.curl.getinfo(pycurl.RESPONSE_CODE),self.response_headers.getvalue(), self.response_body.getvalue()

Java code for the servlet engine:

int postSize = Integer.parseInt(req.getHeader("Content-Length"));
results = new byte[postSize];
int read = 0;
while(read < postSize) {
      int n = req.getInputStream().read(results);
      if (n < 0) break;
      read += n;
  }