python 3.x - Performance issue (long serialization time) with Django rest framework serializers on large dataset like 1000 objec

admin管理员组

文章数量:1026134

I am working on a Django application that uses Django Rest Framework to expose APIs. I am experiencing significant performance issues when serializing large data in Django Rest Framework. I have an endpoint that returns a list of objects e.g. 1000 records. The response is slow and taking around 5-6 seconds. This is not acceptable for the user experience.

Environment Details:

App Engine Runtime: Python 3.9, Django Version: 2.2.28, Django REST Framework Version: 3.12.4 Database : Datastore (NDB)

For example, I have a Test model that has nearly 40 fields. When fetching a list of objects , the response time is about 5-6 seconds which is too slow for my use case.

Here's the relevant code:

Query: 

 return self.ndb_class.all(
            namespace=self.get_namespace(), ancestor=ancestor_key
        )

Code snippet:

        queryset = self.filter_queryset(self.get_queryset())
        page = self.paginate_queryset(queryset)
        if page is not None:
            self.get_related_data(page)
            serializer = self.get_serializer(page, many=True)
            return self.get_paginated_response(serializer.data)

        serializer = self.get_serializer(queryset, many=True)
        results = {'results': serializer.data}
        return Response(results)
 

Profiling Toolbar shows that the query time isn't the issue, but the serialization step seems to be slow. And because of that, I have tested other serialization libraries, including those suggested in online references (such as DRF Serpy, Pydantic, Marshmallow, etc.), but I have not been able to reduce the serialization time to 1 second. It still takes approximately 5+ seconds, more or less.

Any solutions or tips on how to further optimize this or debug the bottleneck would be appreciated!

I am working on a Django application that uses Django Rest Framework to expose APIs. I am experiencing significant performance issues when serializing large data in Django Rest Framework. I have an endpoint that returns a list of objects e.g. 1000 records. The response is slow and taking around 5-6 seconds. This is not acceptable for the user experience.

Environment Details:

App Engine Runtime: Python 3.9, Django Version: 2.2.28, Django REST Framework Version: 3.12.4 Database : Datastore (NDB)

For example, I have a Test model that has nearly 40 fields. When fetching a list of objects , the response time is about 5-6 seconds which is too slow for my use case.

Here's the relevant code:

Query: 

 return self.ndb_class.all(
            namespace=self.get_namespace(), ancestor=ancestor_key
        )

Code snippet:

        queryset = self.filter_queryset(self.get_queryset())
        page = self.paginate_queryset(queryset)
        if page is not None:
            self.get_related_data(page)
            serializer = self.get_serializer(page, many=True)
            return self.get_paginated_response(serializer.data)

        serializer = self.get_serializer(queryset, many=True)
        results = {'results': serializer.data}
        return Response(results)
 

Profiling Toolbar shows that the query time isn't the issue, but the serialization step seems to be slow. And because of that, I have tested other serialization libraries, including those suggested in online references (such as DRF Serpy, Pydantic, Marshmallow, etc.), but I have not been able to reduce the serialization time to 1 second. It still takes approximately 5+ seconds, more or less.

Any solutions or tips on how to further optimize this or debug the bottleneck would be appreciated!

本文标签：