Optimizing django rest framework API

Question

I have created an API to get all the data from by database. There is a "deployment" table and a related table "sensors" which has a foreign key referencing the deployment, a one-many relationship. The serializer creates a nested JSON. Currently if I request all records it takes roughly 45 seconds to return the data (17,000 JSON lines).

How do I profile my Django application to determine what is the bottleneck?

Any suggestions on what can be improved to speed this up? Or is this is as good as it's going to get?

models.py

class deployment(models.Model):
    ADD_DATE = models.DateTimeField() #creation of record in db
    #...30 more fields....

class sensor(models.Model): 

    DEPLOYMENT = models.ForeignKey(deployment, related_name='sensors', on_delete=models.CASCADE)

    ADD_DATE = models.DateTimeField() #creation of record in db
    SENSOR = models.ForeignKey(sensor_types, to_field="VALUE", max_length=50, on_delete=models.PROTECT)
    #...5 more foreign key fields...

views.py

class GetCrtMetadata(generics.ListAPIView): #Read only
    serializer_class = CurrentDeploymentSerializer
    queryset=deployment.objects.all().prefetch_related("sensors")
    filter_backends = [DjangoFilterBackend]
    filter_fields = [field.name for field in deployment._meta.fields]

deployment app serializers.py

class CurrentDeploymentSerializer(serializers.ModelSerializer):
    #Returns deployment with sensors
    sensors = SensorSerializer(many=True)

    class Meta:
        model = deployment

        fields = [field.name for field in deployment._meta.fields]
        fields.extend(['sensors'])
        read_only_fields = fields

sensor app serializers.py

class SensorSerializer(serializers.ModelSerializer):
    class Meta:
        model = sensor
        fields = [field.name for field in sensor._meta.fields]

You can try to prepare json manually (plain lists and dicts, without using drf serializers), this will be probably close to the best performance. You can also check actual sql to make sure you're not making extra queries. You can use cProfile to identify slow parts. And, yeah, serving large json will be slow in general, so it would be good if you can paginate it in some way — awesoon
– awesoon, Commented Jun 17, 2021 at 18:03

Yousef Alm · Accepted Answer · 2021-06-17 17:59:02Z

1

Why requesting all data at once?

Try to use Pagination to get that data you only need and request more if needed.

answered Jun 17, 2021 at 17:59

Yousef Alm

2381 silver badge8 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

R. Anderson Over a year ago

This API will be used to get data into matlab or a local pandas environment for further data analysis. I don't know the exact use cases yet, so I want to be able to send the entire data set. But it's likely the user will want to filter first (thus the filter backend code) and then only get back certain columns (still need to implement this, and would welcome suggestions). Other suggestions for transferring a large amount of data are also welcome.

Collectives™ on Stack Overflow

Optimizing django rest framework API

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related