.net 5 performance of Dictionary vs List vs other in large iteration

Question

I have an algorithm that works with 80,000 items and a dictionary, but the performance is not so good, so I'm looking at a way to optimize it, maybe I'm using the wrong list type?

The scenario, I have 80,000 map references that I need to group, based on how 'close' they are. If they're within X distance, then they get added to a group, or they're a new reference point.

Currently I use a Dictionary with the key = map reference, value = a list of references that are close (within X distance).

For 80,000 objects, that a lot of iterating and it takes minutes to complete.

Any thought so on this ?

Pseudo code.

dictionary (reference, list<references>) referenceGroup = new...

foreach (reference in references) {
    foreach (group in referenceGroup) {
        if (reference.distancefrom(group.key) <= X) {
            referenceGroup.value.add(reference)
            break
        }
    }
    if (neverAddedToGroup) {
        referenceGroup.add(reference, reference)
    }
}