Querying Django model when max_allowed_packet in MySQL is exceeded

920 Views Asked by At

So I've got this periodic task of sending an automated report to the user every month. The problem I encountered while generating the report data was that the MySQL DB has tons of report data for each user, so when I try to query on the User model, I get OperationalError: (1153, "Got a packet bigger than 'max_allowed_packet' bytes").

I've gone into the dbshell and check what the setting for that variable is, and it's the max allowed value (1 GB).

So I'm basically stuck here. Is there any way to get all the data without hitting that OperationalError?

The code is as follows (I've put in dummy names as I can't reveal company information) -

user_ids = list(Model1.objects.filter(param=param_value).values_list('user_id', flat=True)) # returns 143992 user_ids
users = User.objects.filter(user_id__in=user_ids)

I then try to iterate over users, but I hit the OperationalError.

I've also tried to split up the queryset like so -

slices = []
step = 1000
while True:
    sliced_queryset = users[step-1000:step]
    slices.append(sliced_queryset)
    step += 1000
    if sliced_queryset.count() < 1000:
        break

But I hit the same error for .count().

0

There are 0 best solutions below