Why can't get a connection to Cassandra running on Docker from a Spring Boot instace using spring-boot-starter-data-cassandra on first boot?

Question

I have the following

  cassandra:
    image: cassandra:latest
    ports:
      - 9042:9042
    volumes:
      - ./cassandra/image:/var/lib/cassandra
    environment:
      - CASSANDRA_AUTHENTICATOR=AllowAllAuthenticator
      - CASSANDRA_AUTHORIZER=AllowAllAuthorizer

The cassandra instance networking looks like this...

"Networks": {
            "cbusha-infra_default": {
                "IPAMConfig": null,
                "Links": null,
                "Aliases": [
                    "cbusha-infra-cassandra-1",
                    "cassandra",
                    "572f0770b41e"
                ],
                "MacAddress": "02:42:ac:1a:00:04",
                "NetworkID": "b44e49f0f195651a259b7b859fcadda128d359db18de4ab0a4e8b3efa4ed0e35",
                "EndpointID": "6d5fb1b98d2c427a760030a4804db29798893517b48616409575babe0f0f9ae8",
                "Gateway": "172.26.0.1",
                "IPAddress": "172.26.0.4",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "DriverOpts": null,
                "DNSNames": [
                    "cbusha-infra-cassandra-1",
                    "cassandra",
                    "572f0770b41e"
                ]
            }
        }

I confirm I can connect locally using the IJ connection manager

I am now trying to connect so I have the following config from my spring app

spring:
  cassandra:
    contact-points: cassandra
    port: 9042
    keyspace-name: cbusha
    local-datacenter: datacenter1
    schema-action: CREATE_IF_NOT_EXISTS
    connect-timeout-millis: 30000 # 30 seconds
    read-timeout-millis: 30000 # 30 seconds

and I even use a function to make sure it is up first (I also added logic to test the keyspace is available as well)...

@SpringBootApplication
public class BackendApplication {

    private static final Logger log = LoggerFactory.getLogger(BackendApplication.class);

    public static void main(String[] args) {
        waitForCassandra();
        SpringApplication sa = new SpringApplication(BackendApplication.class);
        sa.addBootstrapRegistryInitializer(new MyBootstrapInitializer());
        sa.run(args);
    }

    private static void waitForCassandra() {
        CqlSessionBuilder builder = CqlSession.builder();
        AtomicInteger attempts = new AtomicInteger();
        try (ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor()) {
            executor.scheduleAtFixedRate(() -> {
                try (CqlSession session = builder.build()) {
                    ResultSet rs = session.execute("SELECT keyspace_name FROM system_schema.keyspaces WHERE keyspace_name = 'cbusha';");
                    if (rs.one() != null) {
                        executor.shutdown();
                    } else {
                        if (attempts.incrementAndGet() >= 12) { // 12 attempts * 10 seconds sleep = 2 minutes
                            log.error("Keyspace cbusha does not exist - exiting after 2 minutes");
                            System.exit(1);
                        }
                        log.debug("Keyspace cbusha does not exist - sleeping");
                    }
                } catch (Exception e) {
                    if (attempts.incrementAndGet() >= 12) { // 12 attempts * 10 seconds sleep = 2 minutes
                        log.error("Cassandra is unavailable - exiting after 2 minutes");
                        System.exit(1);
                    }
                    log.debug("Cassandra is unavailable - sleeping");
                }
            }, 0, 10, TimeUnit.SECONDS);
        }
        log.info("Cassandra is up - executing command");
    }
}

But when I try to start on docker I see the connection is accessible

2024-03-24 13:33:45 17:33:45.055 [main] INFO com.cbusha.be.BackendApplication -- Cassandra is up - executing command

However, further down when it tries to create the channel I get the errors in this gist.

I see it is using the proper profile and the docker dns is resolving here endPoint=cassandra/172.26.0.4:9042

If I restart the container a little later it works.

Why is this and is there a way to confirm the channel will work before starting similar to the connection?

Answer 1

I suspect that this is because you have the initialisation happening in a separate service (s). Your code is not waiting for initialisation to complete before connecting to Cassandra.

Your code needs to wait for Cassandra to be available before connecting. See your previous question here where the initialisation script also needed to wait to make a connection.

├── docker-compose.yml
├── Dockerfile
├── pom.xml
├── src
│   └── main
│       ├── java
│       │   └── com
│       │       └── example
│       │           └── demo
│       │               ├── config
│       │               │   └── CassandraConnectionCheck.java
│       │               └── DemoApplication.java
│       └── resources
│           └── application.properties
└── wait-for-cassandra.sh

CassandraConnectionCheck.java

package com.example.demo.config;

import com.datastax.oss.driver.api.core.CqlSession;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.CommandLineRunner;
import org.springframework.stereotype.Component;

@Component
public class CassandraConnectionCheck implements CommandLineRunner {
    private static final Logger logger = LoggerFactory.getLogger(CassandraConnectionCheck.class);

    @Value("${spring.data.cassandra.contact-points}")
    private String cassandraHost;
    @Value("${spring.data.cassandra.port}")
    private int cassandraPort;

    private final CqlSession cqlSession;
    public CassandraConnectionCheck(CqlSession cqlSession) {
        this.cqlSession = cqlSession;
    }

    @Override
    public void run(String... args) {
        logger.info(" Making connection to Cassandra at {}:{}.", cassandraHost, cassandraPort);

        if (!cqlSession.getMetadata().getNodes().isEmpty()) {
            logger.info(" Connected to Cassandra at {}:{}.", cassandraHost, cassandraPort);
        } else {
            logger.error(" Failed to connect to Cassandra at {}:{}.", cassandraHost, cassandraPort);
        }
    }
}

application.properties

spring.data.cassandra.keyspace-name=mykeyspace
spring.data.cassandra.contact-points=cassandra
spring.data.cassandra.port=9042
spring.data.cassandra.schema-action=none
spring.data.cassandra.local-datacenter=datacenter1

Dockerfile

FROM maven:3.6.3-openjdk-11-slim AS build

WORKDIR /home/app

COPY src src
COPY pom.xml /home/app

RUN mvn clean package -DskipTests

FROM openjdk:11-jre-slim

RUN apt-get update && apt-get install -y netcat && apt-get clean

COPY --from=build /home/app/target/spring-boot-app-0.0.1-SNAPSHOT.jar /usr/local/lib/spring-boot-app.jar

EXPOSE 9000

COPY wait-for-cassandra.sh .
RUN chmod +x wait-for-cassandra.sh

ENTRYPOINT ["./wait-for-cassandra.sh"]

wait-for-cassandra.sh (This waits for Cassandra to be available, then creates the keyspace and finally runs the Spring Boot application.)

#!/bin/bash

CASSANDRA_HOST=cassandra
CASSANDRA_PORT=9042
MAX_RETRIES=10
RETRY_INTERVAL=10

echo "Waiting for Cassandra at ${CASSANDRA_HOST}:${CASSANDRA_PORT} to be ready..."

success=false
for ((i=1;i<=MAX_RETRIES;i++)); do
    if nc -z $CASSANDRA_HOST $CASSANDRA_PORT; then
        echo "Cassandra is ready."
        success=true
        break
    else
        echo "Cassandra is not ready yet. Attempt ${i}/${MAX_RETRIES}."
        sleep $RETRY_INTERVAL
    fi
done

if [ "$success" = true ] ; then
    echo "Create keyspace..."
    cqlsh -e "CREATE KEYSPACE IF NOT EXISTS mykeyspace WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'};" cassandra 9042
    echo "Done!"

    echo "Start Spring Boot application."
    exec java -jar /usr/local/lib/spring-boot-app.jar
else
    echo "Failed to connect to Cassandra after ${MAX_RETRIES} attempts."
fi

Answer 2

Overriding the auto conf bean ended up working for me...

@Configuration
public class CassandraConfig {
    private static final Logger log = LoggerFactory.getLogger(CassandraConfig.class);
    private static final CountDownLatch latch = new CountDownLatch(1);
    private static CqlSession session;

    private static void waitForCassandra() {
        CqlSessionBuilder builder = CqlSession.builder();
        AtomicInteger attempts = new AtomicInteger();
        try (ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor()) {
            executor.scheduleAtFixedRate(() -> {
                try {
                    session = builder.build();
                    ResultSet rs = session.execute("SELECT keyspace_name FROM system_schema.keyspaces WHERE keyspace_name = 'cbusha';");
                    if (rs.one() != null) {
                        executor.shutdown();
                        latch.countDown(); // Reduce the count, releasing the latch
                    } else {
                        handleFailure(attempts, "Keyspace cbusha does not exist");
                    }
                } catch (Exception e) {
                    handleFailure(attempts, "Cassandra is unavailable");
                }
            }, 0, 10, TimeUnit.SECONDS);
        }
    }

    private static void handleFailure(AtomicInteger attempts, String errorMessage) {
        if (attempts.incrementAndGet() >= 12) { // 12 attempts * 10 seconds sleep = 2 minutes
            log.error(errorMessage + " - exiting after 2 minutes");
            System.exit(1);
        }
        log.debug(errorMessage + " - sleeping");
    }

    @Bean
    public CqlSession session() throws InterruptedException {
        waitForCassandra();
        latch.await(); // This will block until session is not null
        log.info("Cassandra is up");
        return session;
    }
}

Because org.springframework.boot:spring-boot-starter-data-cassandra uses this session bean it blocks the library from getting bootstrapped before it is ready. Now everything boots correctly