Realtime complex Geoquery within Firebase for 100K users

Question

The use case:

• 50K to 100K trucks updating both location and internal engine info in real time.
• From a console, it's crucial to update the interface every time a truck within some range changes location OR changes internal engine info. So if location is the same but engine info is new, should trigger update.

The problem:

• Geofire only queries by location, so I'd have to create a Geoquery AND listen to each truck's info individually, which is impracticable ad expensive both in Realtime DB and Firestore.
• Firestore is not clear about real time listeners combined with Geoqueries(as far as I know there is no onSnapshot for Geoqueries in Firestore and even so would be super expensive for this case).

The suboptimal alternative: [Web Javascript Firebase V9 Modular] I would use Firestore, create a Geoquery for a given range(which is static, not real time) and get all docs from it. Then in the console the user would only see real time info by clicking into a specific truck(document), so I could attach onSnapshot to that document.

Is this use case simply not supported or is there a way to model data to accommodate it at a reasonable cost ?

Answer 1

GeoFire is only fully supported on the Realtime Database. GeoFire's data is intended to be kept separate from your own data and linked to it by a unique key/id. This minimises the indexing required to execute queries and minimises the data handed around while querying.

There is a workaround for Cloud Firestore but it's not ideal for your use case because it grabs all the nearby records then filters the rest on the client.

If you are curious, a GeoFire entry looks similar to the following when uploaded (this is formatted for readability):

"path/to/geofire/": {
  "<key>/": {
    ".priority": string, // a geohash, hidden
    "g": string, // a geohash
    "l": {
      "0": number, // latitude, number
      "1": number, // longitude, number
    }
  }
}

As can be seen above, there isn't any user-provided data here only a "key". This key can have any meaning such as the code of an airport in a registry, a push ID under a Realtime Database location, a Cloud Firestore document ID or some base64-encoded data.

With the introduction of Firestore, a number of users store the GeoFire data in the Realtime Database and link it back to Cloud Firestore documents using the key stored in GeoFire.

In the below example, a truckInfo object looks like this:

interface TruckInfo {
  key: string;
  location: Geopoint; // [number, number]
  distance: number;
  snapshot: DataSnapshot | undefined;
  cancelling?: number; // an ID from a setTimeout call
  errorCount: number; // number of errors trying to attach data listener
  hidden?: true; // indicates whether the vehicle is currently out of range
  listening?: boolean; // indicates whether the listener was attached successfully
}

For the below code to work, you must define two callback methods:

const truckUpdatedCallback = (truckInfo, snapshot) => { // or you can use ({ key, location, snapshot }) => { ... }
  // TODO: handle new trucks, updated locations and/or updated truck info
  // truckInfo.hidden may be true!
}

const truckRemovedCallback = (truckInfo, snapshot) => {
  // TODO: remove completely (deleted/out-of-range)
}

These callbacks are then invoked by the following "engine":

const firebaseRef = ref(getDatabase(), "gf"), // RTDB location for GeoFire data
  trucksColRef = collection(getFirestore(), "trucks"), // Firestore location for truck-related data
  geoFireInstance = new geofire.GeoFire(firebaseRef),
  trackedTrucks = new Map(), // holds all the tracked trucks
  listenToTruck = (truckInfo) => { // attaches the Firestore listeners for the trucks
    if (truckInfo.cancelling !== void 0) {
      clearTimeout(truckInfo.cancelling);
      delete truckInfo.cancelling;
    }
   
    if (truckInfo.listening || truckInfo.errorCount >= 3)
      return; // do nothing.

    truckInfo.unsub = onSnapshot(
      truckInfo.docRef,
      (snapshot) => {
        truckInfo.listening = true;
        truckInfo.errorCount = 0;
        truckInfo.snapshot = snapshot;
        truckUpdatedCallback(truckInfo, snapshot); // fire callback
      },
      (err) => {
        truckInfo.listening = false;
        truckInfo.errorCount++;
        console.error("Failed to track truck #" + truckInfo.key, err);
      }
    )
  },
  cancelQuery = () => { // removes the listeners for all trucks and disables query
    // prevents all future updates
    geoQuery.cancel();
    trackedTrucks.forEach(({unsub}) => {
      unsub && unsub();
    });
  };

const geoQuery = geoFireInstance.query({ center, radius });

geoQuery.on("key_entered", function(key, location, distance) {
  let truckInfo = trackedTrucks.get(key);

  if (!truckInfo) {
    // new truck to track
    const docRef = document(trucksColRef, key);
    truckInfo = { key, location, distance, docRef, errorCount: 0 };
    trackedTrucks.set(key, truckInfo);
  } else {
    // truck has re-entered watch area, update position
    Object.assign(truckInfo, { location, distance });
    delete truckInfo.hidden;
  }

  listenToTruck(truckInfo);
});

geoQuery.on("key_moved", function(key, location, distance) {
  const truckInfo = trackedTrucks.get(key);
  if (!truckInfo) return; // not being tracked?
  Object.assign(truckInfo, { location, distance });
  truckUpdatedCallback(truckInfo, snapshot); // fire callback
});

geoQuery.on("key_exited", function(key, location, distance) {
  const truckInfo = trackedTrucks.get(key);
  if (!truckInfo) return; // not being tracked?

  truckInfo.hidden = true;

  const unsub = truckInfo.unsub,
    cleanup = () => { // removes any listeners for this truck and removes it from tracking
      unsub && unsub();
      truckInfo.listening = false;
      trackedTrucks.delete(key);
      truckRemovedCallback(truckInfo, snapshot); // fire removed callback
    };

  if (location === null && distance === null) {
    // removed from database, immediately remove from view
    return cleanup();
  }

  // keep getting updates for at least 60s after going out of
  // range in case vehicle returns. Afterwards, remove from view
  truckInfo.cancelling = setTimeout(cleanup, 60000);

  // fire callback (to hide the truck)
  truckUpdatedCallback(truckInfo, snapshot);
});