In Scala 2.13+, how can one efficiently implement a conversion to a particular standard collection type?

Question

The current collection framework favors the use of to method in order to convert to the target collection type, and with an implicit conversion available from various collection companion objects to the Factory argument, this creates a neat, uniform interface. It unfortunately makes very hard optimisations which were easy in the old framework with CanBuildFrom. Lets say I have a custom collection type Unique[T], which is a combination of Set and Seq in that the order in which elements follows the order of insertion, but any element can occur only once. Because it offers fast indexOf, apply(i :Int) and contains, conversions to both Set and Seq can be O(1) with a simple wrapper. I can override toSet and toSeq, but I see no way of determining inside to (other than extremely questionable reflection) that the target factory builds Seq or Set, because the Factory implementation used by the implicit conversion is generic and not a prototype instance like old ReusableCBF.