drf custom authentication backend gets executed on the path that doesn't need authentication

Question

I am new to django and I am trying to add permissions from DRF to my project. Ever since I have set DEFAULT_AUTHENTICATION_CLASSES for REST_FRAMEWORK in django settings.py, all the requests are going to the authenticate method of my DEFAULT_AUTHENTICATION_CLASSES irrespective of what permission I set to my view. Later it is coming to my view. So here is the settings I have added:

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': [
        'authentication.customauth.CustomAuthBackend',
    ]
}

And here is my authentication.customauth.CustomAuthBackend:

class CustomAuthBackend(BaseAuthentication):
    def authenticate(self, request):
        user = AuthUtils.get_user_from_token(request)

        if user is None:
            raise AuthenticationFailed('User not found')

        request.user = user
        return user, None

    @staticmethod
    def authenticate_with_password(request):
        email = request.data.get('email')
        role = "CONSUMER" if request.data.get('role') is None else request.data.get('role')
        password = request.data.get('password')

        user = User.objects.filter(email=email, role=role).first()

        if password is not None and user is not None and user.check_password(password):
            return user

The views that actually should be called without authentication have @permission_classes([AllowAny]) permission. Say this login view:

@api_view(['POST'])
@permission_classes([AllowAny])
def login(request):
    user = request.user

    if user and user.is_active:
        serializer = UserSerializer(user)
        tokens_map = AuthUtils.generate_token(request=request, user=user)
        return Response({'success': True, 'user': serializer.data, 'tokens': tokens_map})

    return Response(data={'success': False, 'message': 'User not found'}, status=status.HTTP_404_NOT_FOUND)

With my understanding I think if permission class is rest_framework.permissions.AllowAny no authenticate method should not be called before calling my view.