How to set up Cluster Discovery for a Atomix API application inside kubernetes

Question

I want to use the atomix framework using the java API in my application.

The application should be deployed and scaled via kubernetes. And every Pod should be "connected" with all Pods of the same kubernetes deployment.

I have seen in the documentation that there are different ways how to set up cluster discovery, so that each node of the cluster knows all members but no configuration seems to work for my scenario.

1. Manual config: Manual configure all members in a list will not work for kubernetes.
2. Use multicast: AFAIK multicast is also not a good option in kubernetes.
3. DNS: I also don't think that I could use DNS discovery for it (because DNS is normally per service and not per Pod)

There is also a section about kubernetes deployment in the atomix manual but it seems that this is only useful for launching multiple atomix agents in a cluster and not for scaling a custom application which uses the Atomix API (please let me know if I get this wrong)

I don't have found any examples for such a setting even if it should be quite a common task to solve...

Answer 1

You can use DNS provided you configure a service specifically for this task. In k8s, every pod can be a member of an arbitrary number of services (because a service is just a load-balancer). So you could define a service for this purpose

---
apiVersion: v1
kind: Service
metadata:
  labels:
    - myLabel
  name: service-clustering
spec:
  clusterIP: None
  publishNotReadyAddresses: true
  ports:
    - name: appName
      port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    - matchLabel
  type: ClusterIP

Important here is the publishNotReadyAddresses, because you want to prevent a split-brain scenario during bootup before all pods passed their readiness checks. Afterwards you can just use DNS to discover the individual pods (using dnsjava):

    private Stream<String> getPodUris() {
        return Optional.ofNullable(new Lookup("service-clustering", Type.SRV))
            .stream()
            .flatMap(Arrays::stream)
            .filter(r -> r instanceof SRVRecord)
            .map(r -> ((SRVRecord) r).getTarget().toString());
    }

For dynamic scaling you'd need to repeat this query in a scheduled task every couple of seconds to inform Atomix about membership changes.