Django REST Framework caching bug

Question

TL;DR

I am looking for a way to clear the cache after a request, or completely disable it when running tests. Django REST Framework seems to cache the results and I need a way around this.

Long version and code

Well this turned out to behave very weird as I kept testing it. In the end, I got it to work, but I really don't like my workaround and in the name of knowledge, I have to find out why this happens and how to solve this problem properly.

So, I have an APITestCase class declared like this:

class UserTests(APITestCase):

Inside this class, I have a test function for my user-list view, as I have a custom queryset depending on the permissions. To clear things up:

a superuser can get the whole users list (4 instances returned),
staff members cannot see superusers (3 instances returned),
normal users can only get 1 result, their own user (1 instance returned)

The test function version that works:

def test_user_querysets(self):

    url = reverse('user-list')

    # Creating a user
    user = User(username='user', password=self.password)
    user.set_password(self.password)
    user.save()

    # Creating a second user
    user2 = User(username='user2', password=self.password)
    user2.set_password(self.password)
    user2.save()

    # Creating a staff user
    staff_user = User(username='staff_user', password=self.password, is_staff=True)
    staff_user.set_password(self.password)
    staff_user.save()

    # Creating a superuser
    superuser = User(username='superuser', password=self.password, is_staff=True, is_superuser=True)
    superuser.set_password(self.password)
    superuser.save()



    # SUPERUSER

    self.client.logout()
    self.client.login(username=superuser.username, password=self.password)

    response = self.client.get(url)

    # HTTP_200_OK
    self.assertEqual(response.status_code, status.HTTP_200_OK)

    # All users contained in list
    self.assertEqual(response.data['extras']['total_results'], 4)



    # STAFF USER

    self.client.logout()
    self.client.login(username=staff_user.username, password=self.password)

    response = self.client.get(url)

    # HTTP_200_OK
    self.assertEqual(response.status_code, status.HTTP_200_OK)

    # Superuser cannot be contained in list
    self.assertEqual(response.data['extras']['total_results'], 3)



    # REGULAR USER

    self.client.logout()
    self.client.login(username=user2.username, password=self.password)

    response = self.client.get(url)

    # HTTP_200_OK
    self.assertEqual(response.status_code, status.HTTP_200_OK)

    # Only 1 user can be returned
    self.assertEqual(response.data['extras']['total_results'], 1)

    # User returned is current user
    self.assertEqual(response.data['users'][0]['username'], user2.username)

As you see, I am testing user permissions in this order: superuser, staff, normal user. And this works, so...

Funny thing:

If I change the order of the tests, and start with normal user, staff, superuser, the tests fail. The response from the first request gets cached, and then I get the same response when I log in as staff user, so the number of results is again 1.

The version that doesn't work:

it's exactly the same as before, only the tests are made in reverse order

def test_user_querysets(self):

    url = reverse('user-list')

    # Creating a user
    user = User(username='user', password=self.password)
    user.set_password(self.password)
    user.save()

    # Creating a second user
    user2 = User(username='user2', password=self.password)
    user2.set_password(self.password)
    user2.save()

    # Creating a staff user
    staff_user = User(username='staff_user', password=self.password, is_staff=True)
    staff_user.set_password(self.password)
    staff_user.save()

    # Creating a superuser
    superuser = User(username='superuser', password=self.password, is_staff=True, is_superuser=True)
    superuser.set_password(self.password)
    superuser.save()



    # REGULAR USER

    self.client.logout()
    self.client.login(username=user2.username, password=self.password)

    response = self.client.get(url)

    # HTTP_200_OK
    self.assertEqual(response.status_code, status.HTTP_200_OK)

    # Only 1 user can be returned
    self.assertEqual(response.data['extras']['total_results'], 1)

    # User returned is current user
    self.assertEqual(response.data['users'][0]['username'], user2.username)



    # STAFF USER

    self.client.logout()
    self.client.login(username=staff_user.username, password=self.password)

    response = self.client.get(url)

    # HTTP_200_OK
    self.assertEqual(response.status_code, status.HTTP_200_OK)

    # Superuser cannot be contained in list
    self.assertEqual(response.data['extras']['total_results'], 3)



    # SUPERUSER

    self.client.logout()
    self.client.login(username=superuser.username, password=self.password)

    response = self.client.get(url)

    # HTTP_200_OK
    self.assertEqual(response.status_code, status.HTTP_200_OK)

    # All users contained in list
    self.assertEqual(response.data['extras']['total_results'], 4)

I am working in python 2.7 with the following package versions:

Django==1.8.6
djangorestframework==3.3.1
Markdown==2.6.4
MySQL-python==1.2.5
wheel==0.24.0

UPDATE

I am using the default django cache, meaning I haven't put anything about cache in the django settings.

As suggested, I tried disabling the default Django cache:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
    }
}

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.SessionAuthentication',
    )
}

The problem stands on.

Even though I don't think the problem is located here, this is my UserViewSet:

api.py (the important part)

class UserViewSet(  
    mixins.RetrieveModelMixin, 
    mixins.UpdateModelMixin,
    mixins.ListModelMixin,
    viewsets.GenericViewSet
):
    queryset = User.objects.all()
    serializer_class = UserExpenseSerializer
    permission_classes = (IsAuthenticated, )
    allowed_methods = ('GET', 'PATCH', 'OPTIONS', 'HEAD')

    def get_serializer_class(self):
        if self.action == 'retrieve':
            return UserExpenseSerializer
        return UserSerializer

    def get_queryset(self):
        if(self.action == 'list'):
            return User.objects.all()
        if self.request.user.is_superuser:
            return User.objects.all()
        if self.request.user.is_staff:
            return User.objects.exclude(is_superuser=True)
        return User.objects.filter(pk = self.request.user.id)

    def list(self, request):
        filter_obj = UsersFilter(self.request)
        users = filter_obj.do_query()
        extras = filter_obj.get_extras()
        serializer = UserSerializer(users, context={'request' : request}, many=True)
        return Response({'users' : serializer.data, 'extras' : extras}, views.status.HTTP_200_OK)

filters.py

class UsersFilter:
    offset = 0
    limit = 50
    count = 0
    total_pages = 0
    filter_params = {}

    def __init__(self, request):

        if not request.user.is_superuser:
            self.filter_params['is_superuser'] = False

        if (not request.user.is_superuser and not request.user.is_staff):
            self.filter_params['pk'] = request.user.id

        # Read query params
        rpp = request.query_params.get('rpp') or 50
        page = request.query_params.get('page') or 1
        search_string = request.query_params.get('search')

        # Validate

        self.rpp = int(rpp) or 50
        self.page = int(page) or 1

        # Set filter
        set_if_not_none(self.filter_params, 'username__contains', search_string)

        # Count total results

        self.count = User.objects.filter(**self.filter_params).count()
        self.total_pages = int(self.count / self.rpp) + 1

        # Set limits
        self.offset = (self.page - 1) * self.rpp
        self.limit = self.page * self.rpp

    def get_filter_params(self):
        return self.filter_params

    def get_offset(self):
        return self.offset

    def get_limit(self):
        return self.limit

    def do_query(self):
        users = User.objects.filter(**self.filter_params)[self.offset:self.limit]
        return users

    def get_query_info(self):
        query_info = {
            'total_results' : self.count,
            'results_per_page' : self.rpp,
            'current_page' : self.page,
            'total_pages' : self.total_pages
        }
        return query_info

UPDATE 2

As Linovia pointed out, the problem was not cache or any other DRF problem, but the filter. Here's the fixed filter class:

class UsersFilter:

    def __init__(self, request):

        self.filter_params = {}
        self.offset = 0
        self.limit = 50
        self.count = 0
        self.total_pages = 0
        self.extras = {}

        if not request.user.is_superuser:
        # and so long...

If self.client is a django.test.Client and you're interacting as a different user, why not create a new Client()? This should more closely match the model of what is happening anyways. A Client instance is a stand alone browser interaction with your site. Perhaps that is why Django is getting confused. — Ross Rogers
– Ross Rogers, Commented Nov 20, 2015 at 15:46
I actually did try that, the same problem appears. It has something to do with the DRF caching. Another funny thing is that if I test the staff user before the normal user, the test passes. Blows my mind. — Skatch
– Skatch, Commented Nov 20, 2015 at 15:53
Are you sure you're limiting your results from the url you're querying? What does your queryset parameter or get_queryset() function look like on that view? — Ross Rogers
– Ross Rogers, Commented Nov 20, 2015 at 15:57
Completely sure, checked it like 50 times in the browsable api :). I'll paste the queryset now. — Skatch
– Skatch, Commented Nov 20, 2015 at 16:02
As a matter of fact I can't do that, it also includes a filter function for the data that customizes the queryset with several query_params. However, this part works as I have tested it a lot of times and the application actually works as I want it to. And also, as I said, the test works if I make a request for a staff_user first :))) — Skatch
– Skatch, Commented Nov 20, 2015 at 16:05

Linovia · Accepted Answer · 2015-11-27 14:40:29Z

4

+50

Actually you create a new user which should make 2 users and you assert the length against 3. Not going to work even without caching.

Edit: So you actually have you issue because the use of mutables objects at the class level.

Here's the evil code:

class UsersFilter:
    filter_params = {}

    def __init__(self, request):
        if not request.user.is_superuser:
            self.filter_params['is_superuser'] = False

Which should actually be:

class UsersFilter:
    def __init__(self, request):
        filter_params = {}
        if not request.user.is_superuser:
            self.filter_params['is_superuser'] = False

Otherwise UsersFilter.filter_params will be kept from one request to another and never resets. See http://www.toptal.com/python/python-class-attributes-an-overly-thorough-guide for more details about this.

edited Nov 27, 2015 at 14:40

answered Nov 20, 2015 at 15:51

Linovia

21.1k4 gold badges53 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

Skatch Over a year ago

That is not the problem, I didn't put the entire code... There are 2 normal users created before this, and this code snippet starts after that. The problem is not some minor error, I know it is the caching after 2 hours of testing and trying different methods. I will update the snippet right away.

Skatch Over a year ago

Question updated, you can take a look now if you want to.

Linovia Over a year ago

Strange behavior. This feels like you've set the query set on the module or class level and filter after that which would work for first example because it'll have all the users and not in the second (only one item for the QS).

Linovia Over a year ago

It was working because you went from the widest filter to the smallest one. Therefore every new query was "hiding" the previous one

Skatch Over a year ago

Ah, yes, I feel so dumb now :)

|

Panda · Accepted Answer · 2015-11-27 12:36:39Z

0

You can disable caching in debug mode by adding this to your settings.py

if DEBUG:
    CACHES = {
        'default': {
            'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
        }
    }

https://docs.djangoproject.com/en/dev/topics/cache/?from=olddocs/#dummy-caching-for-development

You can then disable caching by toggling DEBUG in settings.py or by having separate develop/test and /deploy settings.py files.

If you want to avoid separate files or toggling, then you can set DEBUG to true for certain tests with the override_settings decorator:

from django.test.utils import override_settings
from django.test import TestCase
from django.conf import settings

class MyTest(TestCase):
    @override_settings(DEBUG=True)
    def test_debug(self):
        self.assertTrue(settings.DEBUG)

edited Nov 27, 2015 at 12:36

answered Nov 27, 2015 at 9:54

Panda

1,23918 silver badges28 bronze badges

4 Comments

Skatch Over a year ago

Yeah, that's in the docs, but how can I disable it only when I'm running tests without editing the settings.py file every time?

Panda Over a year ago

You could add if 'test' in sys.argv: so this will only be defined when running python manage.py test. In the past I've also used if DEBUG:.

Skatch Over a year ago

if DEBUG sounds like a good solution, can you please provide a more elaborate answer with exact code sample, how to change cache engine inside code execution, so I can test and accept it?

Skatch Over a year ago

Upvote for this info, but unfortunately it doesn't solve the problem (I even omitted the if clause, just disabled cache). I guess this doesn't have anything to do with Django, but some bug in DRF. I'm reading the docs over and over, and there is nothing on disabling cache, only those throttling classes that seem disabled by default. So weird...

Collectives™ on Stack Overflow

Django REST Framework caching bug

TL;DR

Long version and code

The test function version that works:

Funny thing:

The version that doesn't work:

UPDATE

UPDATE 2

2 Answers 2

10 Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

TL;DR

Long version and code

The test function version that works:

Funny thing:

The version that doesn't work:

UPDATE

UPDATE 2

2 Answers 2

10 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related