Aggregate of OuterRef in Django Subquery

Question

I have the following two models in Django 4.2:

class Team(Model):
  members = ManyToManyField(...)
  ...

class Size(Model):
  max_members = PositiveSmallIntegerField(blank=True, null=True)
  class Meta:
    ordering = [F('max_members').asc(nulls_last=True)]

The idea is that each Team has a Size but I can't (as far as I know) model this as a SQL relation. But that's OK, we can get the size of a team as follows:

Size.objects.filter(Q(max_members=None) | Q(max__gte=team.members.count())).first()

However if I want to get the size associated with an entire queryset of teams at once, I am stuck. It seems like the following should work:

teams.annotate(
  size=Subquery(
     Size.objects.filter(
       Q(max_members=None) | Q(max_members__gte=Count(OuterRef('members')))
     ).values('pk')[:1]
  )
)

But this produces invalid SQL (we are using PostGres):

SELECT "teams_team".*
       (SELECT U0."id"
        FROM "teams_size" U0
        HAVING (U0."max_members" IS NULL OR U0."max_members" >= (COUNT("teams_teammembers"."user_id")))
        ORDER BY U0."max" ASC NULLS LAST
        LIMIT 1) AS "size"
FROM "teams_team"
         LEFT OUTER JOIN "teams_teammembers" ON ("teams_team"."id" = "teams_teammembers"."team_id")

This is missing a GROUP BY teams_team.id and the HAVING clause should be WHERE.

I can get some of the way there with some hacking: if I swap Count for Func(..., function='COUNT') then Django doesn't realise there is any aggregation and keeps the WHERE clause as it should do. But adding the grouping by hand, by setting qs.query.group_by = ['id'] results in Django automatically also grouping on teams_teammembers.user_id. Hence there is one group for every member of the team, and the count is always 1.

I have seen uses of aggregation within subqueries in django code, but never of an OuterRef so I assume that is what's going wrong here, hence the title.

Ultimately I want to get the IDs of the sizes so that I can match up the actual size objects with the teams as if this were a real relationship. A solution which somehow added a prefetchable django.models.ForeignObject to Team to represent the relationship would actually be even better.

If there is no ORM solution then an idea of how to do this with the least amount of custom SQL within Django would be welcome. Failing that we'll just denormalise the relation but we'd rather not at this point.