Django rest framework parallel execution of many=True field data

Question

I have a serializer, in that I've overridden the validate method.

class ProcessItemSerializer(serializers.Serializer):
    field1 = serializers.CharField(max_length=63)
    # Few more fields

    def validate(self, data):
        # few data processing and HTTP calls
        return data

I have another serializer which use above serializer as many=True field

class DataSerializer(serializers.Serializer):
    items = ProcessItemSerializer(many=True)

If I pass a list of item data to DataSerializer it will process one by one each item. thats desirable!

but if length of items is more than 100 it takes much time. what I want is to execute set of 20-20 items parallel using python-multiprocess so that I can reduce the overall time.

how can I do this in DRF. what method do I have to override?

This isn't a trivial task, at all. Can you explain exactly what type of validation you are doing? It will be much easier to optimize that rather than to attempt any form of distributed processing. — Yuval Adam
– Yuval Adam, Commented Jul 19, 2017 at 9:08
Using multiprocessing for IO bound is not a good idea, you should use coroutine (gevent) — Hadi Farhadi
– Hadi Farhadi, Commented Jul 19, 2017 at 9:17
as @hadi correctly mentioned, your validation is I/O-bound, so multiprocessing isn't going to help you much. — Yuval Adam
– Yuval Adam, Commented Jul 19, 2017 at 9:34
@hadi true, I/O bound but internally that are not dependent . I mean item[0] is not dependent on item[1] at all. — Ashraful Islam
– Ashraful Islam, Commented Jul 19, 2017 at 9:38
If your data does not change frequently, use caching, http cache server — Hadi Farhadi
– Hadi Farhadi, Commented Jul 19, 2017 at 10:04

Michael Rigoni · Accepted Answer · 2017-07-20 12:27:00Z

1

You should be able to process in parallel by overriding ListSerializer and more specifically the to_internal_value method.

So it should look something like:

class ProcessItemListSerializer(ListSerializer):
    def to_internal_value(self, data):
        ....


class ProcessItemSerializer(serializers.Serializer):
    field1 = serializers.CharField(max_length=63)
    # Few more fields
    class Meta:
        list_serializer_class = ProcessItemListSerializer

    def validate(self, data):
        # few data processing and HTTP calls
        return data

In to_internal_value, you may need to copy the beginning of the code till for item in data: and there construct the validation by chunks...

edited Jul 20, 2017 at 12:27

answered Jul 19, 2017 at 10:20

Michael Rigoni

1,97613 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Django rest framework parallel execution of many=True field data

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related