I am using Django rest framework in our application's backend. In one of the api end points we have to send a large data as response. So the orm and serialization functionalities takes about 400mb to send the data to the front end. I am using memory_profiler and when we call another api or the same api it shows 400MB at start and then an additional 400MB is used if the same endpoint is called. So total 800MB. It keeps on adding whenever we call the api till it crashes.
This is my code --->
**views.py **
@api_view(["GET"])
@hasValidToken
def get_all_session_messages(request, user, identity_data, *args, **kwargs):
try:
return chat_services.get_all_session_messages(
user.id,
kwargs["session_id"],
)
except Exception as exc:
return error(message=str(exc))
**services.py **
def get_all_session_messages(user_id, chat_session_id):
try:
query_responses = chat_repository.get_all_query_response_given_chat_id(
chat_session_id
)
return ok(
message="Successfully retrieved all chat session messages",
response_data=query_responses,
)
except Exception as exc:
raise Exception(
"Error retreiving chat session messages in repository " + str(exc)
)
**repository.py**
def get_all_query_response_given_chat_id(
chat_id, limit=None, order_by_created_at=False
):
if not order_by_created_at:
if limit is not None:
query_responses = QueryResponseTracking.objects.filter(chat_id=chat_id)[
:limit
]
else:
query_responses = QueryResponseTracking.objects.filter(chat_id=chat_id)
else:
if limit is not None:
query_responses = QueryResponseTracking.objects.filter(
chat_id=chat_id
).order_by("-created_at")[:limit]
else:
query_responses = QueryResponseTracking.objects.filter(
chat_id=chat_id
).order_by("-created_at")
return QueryResponseTrackingSerializer(query_responses, many=True).data
**serializers.py**
class QueryResponseTrackingSerializer(serializers.ModelSerializer):
class Meta:
model = QueryResponseTracking
fields = "__all__"
**models.py**
class QueryResponseTracking(models.Model):
id = models.UUIDField(default=uuid.uuid4, unique=True, primary_key=True)
chat_id = models.CharField(max_length=200, default="-1")
user_query = models.CharField(max_length=500)
query_meta_data = models.JSONField(default=None, null=True)
response = models.CharField(max_length=500, default=None, null=True)
response_time = models.FloatField(default=None, null=True)
created_at = models.DateTimeField(auto_now_add=True)
feedback = models.BooleanField(default=None, null=True)
We need all the fields in the front end also.
I tried to use gc.collect() after the response in a finally block in views. But it is not working the first time. On the second api request it clears up some memory like 200MB out of 400MB. But it is not consistent. Sometimes just 15 MB gets cleared up. So it keeps on adding the memory with each subsequent request.