Dockerized spring boot app can't access host machine's kafka instance through localhost

Question

I have a dockerized spring-boot web service that needs to produce to a Kafka topic. I'm running Kafka with Zookeeper locally on Ubuntu. My web service is running in Docker on the same machine. I have confirmed with netstat -tlpn that the Kafka broker is active on port 9092.

If I run docker run my-image-name:latest --network="host", the Kafka producer cannot access the bootstrap broker.

KafkaService.java

@Component
public class KafkaService {

    private String bootstrapServerUrl;

    private String topicName;

    private KafkaProducer<String, HandleCaptureKafkaMessage> kafkaProducer;

        // create Producer properties
    public KafkaService(@Value("${kafka-bootstrap-server-url}") String bootstrapServerUrl, @Value("${kafka-topic-name}") String topicName) {
        this.bootstrapServerUrl = bootstrapServerUrl;
        this.topicName = topicName;
        System.out.println("SERVERURL " + bootstrapServerUrl);
        Properties properties = new Properties();
        properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServerUrl);
        properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, HandleCaptureKafkaMessageSerializer.class.getName());

        kafkaProducer = new KafkaProducer<>(properties);
    }

    public void sendMessage(HandleCaptureKafkaMessage handleCaptureKafkaMessage) {
        kafkaProducer.send(new ProducerRecord<>(topicName, null, handleCaptureKafkaMessage), null);
    }

}

Output log

2024-02-05T19:13:00.425Z  INFO 1 --- [           main] o.a.k.clients.producer.ProducerConfig    : ProducerConfig values:
        acks = -1
        auto.include.jmx.reporter = true
        batch.size = 16384
        bootstrap.servers = [localhost:9092]
        buffer.memory = 33554432
...
2024-02-05T19:13:00.498Z  INFO 1 --- [           main] o.a.k.clients.producer.KafkaProducer     : [Producer clientId=producer-1] Instantiated an idempotent producer.
2024-02-05T19:13:00.554Z  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 3.6.1
2024-02-05T19:13:00.559Z  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: 5e3c2b738d253ff5
2024-02-05T19:13:00.560Z  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1707160380540
2024-02-05T19:13:00.830Z  INFO 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Node -1 disconnected.
2024-02-05T19:13:00.832Z  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
2024-02-05T19:13:00.832Z  WARN 1 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected

Any ideas?