68

I have a Django application with a view that accepts a file to be uploaded. Using the Django REST framework I'm subclassing APIView and implementing the post() method like this:

class FileUpload(APIView):
    permission_classes = (IsAuthenticated,)

    def post(self, request, *args, **kwargs):
        try:
            image = request.FILES['image']
            # Image processing here.
            return Response(status=status.HTTP_201_CREATED)
        except KeyError:
            return Response(status=status.HTTP_400_BAD_REQUEST, data={'detail' : 'Expected image.'})

Now I'm trying to write a couple of unittests to ensure authentication is required and that an uploaded file is actually processed.

class TestFileUpload(APITestCase):
    def test_that_authentication_is_required(self):
        self.assertEqual(self.client.post('my_url').status_code, status.HTTP_401_UNAUTHORIZED)

    def test_file_is_accepted(self):
        self.client.force_authenticate(self.user)
        image = Image.new('RGB', (100, 100))
        tmp_file = tempfile.NamedTemporaryFile(suffix='.jpg')
        image.save(tmp_file)
        with open(tmp_file.name, 'rb') as data:
            response = self.client.post('my_url', {'image': data}, format='multipart')
            self.assertEqual(status.HTTP_201_CREATED, response.status_code)

But this fails when the REST framework attempts to encode the request

Traceback (most recent call last):
  File "/home/vagrant/.virtualenvs/myapp/lib/python3.3/site-packages/django/utils/encoding.py", line 104, in force_text
    s = six.text_type(s, encoding, errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 118: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vagrant/webapp/myproject/myapp/tests.py", line 31, in test_that_jpeg_image_is_accepted
    response = self.client.post('my_url', { 'image': data}, format='multipart')
  File "/home/vagrant/.virtualenvs/myapp/lib/python3.3/site-    packages/rest_framework/test.py", line 76, in post
    return self.generic('POST', path, data, content_type, **extra)
  File "/home/vagrant/.virtualenvs/myapp/lib/python3.3/site-packages/rest_framework/compat.py", line 470, in generic
    data = force_bytes_or_smart_bytes(data, settings.DEFAULT_CHARSET)
  File "/home/vagrant/.virtualenvs/myapp/lib/python3.3/site-packages/django/utils/encoding.py", line 73, in smart_text
    return force_text(s, encoding, strings_only, errors)
  File "/home/vagrant/.virtualenvs/myapp/lib/python3.3/site-packages/django/utils/encoding.py", line 116, in force_text
    raise DjangoUnicodeDecodeError(s, *e.args)
django.utils.encoding.DjangoUnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 118: invalid start byte. You passed in b'--BoUnDaRyStRiNg\r\nContent-Disposition: form-data; name="image"; filename="tmpyz2wac.jpg"\r\nContent-Type: image/jpeg\r\n\r\n\xff\xd8\xff[binary data omitted]' (<class 'bytes'>)

How can I make the test client send the data without attempting to decode it as UTF-8?

5
  • 2
    Pass { 'image': file} instead Commented Jun 13, 2014 at 9:19
  • @arocks Eagle eyes! I've corrected the typo in the posting, the actual code did not have this problem. Commented Jun 13, 2014 at 9:26
  • Your code work for me. Thank you! Commented Jul 16, 2014 at 1:45
  • 5
    It was because I was missing format='multipart' - doh Commented Oct 7, 2014 at 19:59
  • where is tempfile coming from? Commented May 11, 2016 at 13:19

5 Answers 5

60

When testing file uploads, you should pass the stream object into the request, not the data.

This was pointed out in the comments by @arocks

Pass { 'image': file} instead

But that didn't full explain why it was needed (and also didn't match the question). For this specific question, you should be doing

from PIL import Image

class TestFileUpload(APITestCase):

    def test_file_is_accepted(self):
        self.client.force_authenticate(self.user)

        image = Image.new('RGB', (100, 100))

        tmp_file = tempfile.NamedTemporaryFile(suffix='.jpg')
        image.save(tmp_file)
        tmp_file.seek(0)

        response = self.client.post('my_url', {'image': tmp_file}, format='multipart')

       self.assertEqual(status.HTTP_201_CREATED, response.status_code)

This will match a standard Django request, where the file is passed in as a stream object, and Django REST Framework handles it. When you just pass in the file data, Django and Django REST Framework interpret it as a string, which causes issues because it is expecting a stream.

And for those coming here looking to another common error, why file uploads just won't work but normal form data will: make sure to set format="multipart" when creating the request.

This also gives a similar issue, and was pointed out by @RobinElvin in the comments

It was because I was missing format='multipart'

Sign up to request clarification or add additional context in comments.

6 Comments

For some reason this leads me to 400 error. The error returned is {"file":["The submitted file is empty."]}.
Note that if you don't want to allocate a tempfile for whatever reason (perf would be one), you can also do tmp_file = BytesIO(b'some text'); this will give you a binary stream that can be passed as a file object. (docs.python.org/3/library/io.html).
Had to add tmp_file.seek(0) before the post, but otherwise perfect! This almost drove me nuts so thanks!
@RudolfOlah Image comes from the Pillow library. See pillow.readthedocs.io/en/stable.
It is important to pay attention that it is format=multipart and not content-type. I will leave this here if someone is also making the same mistake and getting a 415
|
22

Python 3 users: make sure you open the file in mode='rb' (read,binary). Otherwise, when Django calls read on the file the utf-8 codec will immediately start choking. The file should be decoded as binary not utf-8, ascii or any other encoding.

# This won't work in Python 3
with open(tmp_file.name) as fp:
        response = self.client.post('my_url', 
                                   {'image': fp}, 
                                   format='multipart')

# Set the mode to binary and read so it can be decoded as binary
with open(tmp_file.name, 'rb') as fp:
        response = self.client.post('my_url', 
                                   {'image': fp}, 
                                   format='multipart')

2 Comments

I think that {'image': data} ought to be {'image': fp} in the answer. I was struggling with uploads until I found this post, yet my tests did not pass until I put the file handle object fp in place of data in the {'image': data} dictionary described above. ({'image': fp} worked in my case, {'image': data} did not.)
Updated. Thanks DMfll.
15

You can use Django built-in SimpleUploadedFile:

from django.core.files.uploadedfile import SimpleUploadedFile

class TestFileUpload(APITestCase):
    ...

    def test_file_is_accepted(self):
        ...

       tmp_file = SimpleUploadedFile(
                      "file.jpg", "file_content", content_type="image/jpg")

       response = self.client.post(
                      'my_url', {'image': tmp_file}, format='multipart')
       self.assertEqual(response.status_code, status.HTTP_201_CREATED)

Comments

9

It's not so simple to understand how to do it if you want to use the PATCH method, but I found the solution in this question.

from django.test.client import BOUNDARY, MULTIPART_CONTENT, encode_multipart

with open(tmp_file.name, 'rb') as fp:
    response = self.client.patch(
        'my_url', 
        encode_multipart(BOUNDARY, {'image': fp}), 
        content_type=MULTIPART_CONTENT
    )

1 Comment

Arrrgh, thank you so much, this and adding the proper parsers to the ViewSet allowed me to test a self.client.PUT call that seemed impossible!
2

For those in Windows, the answer is a bit different. I had to do the following:

resp = None
with tempfile.NamedTemporaryFile(suffix='.jpg', delete=False) as tmp_file:
    image = Image.new('RGB', (100, 100), "#ddd")
    image.save(tmp_file, format="JPEG")
    tmp_file.close()

# create status update
with open(tmp_file.name, 'rb') as photo:
    resp = self.client.post('/api/articles/', {'title': 'title',
                                               'content': 'content',
                                               'photo': photo,
                                               }, format='multipart')
os.remove(tmp_file.name)

The difference, as pointed in this answer (https://stackoverflow.com/a/23212515/72350), the file cannot be used after it was closed in Windows. Under Linux, @Meistro's answer should work.

2 Comments

On OSX10.12.5 I got ValueError: I/O operation on closed file
I don't think you need the tmp_file.close() with the with statement.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.