How to use PulsarIO.write from Apache Beam in pipeline (Missing required properties: clientUrl)

Question

I have a beam pipeline where I want to write the output to a pulsar topic.

In the end I say

    pCollection.apply("Send to Pulsar", PulsarIO.write().withClientUrl(pulsarClientUrl).withTopic(pulsarTopic));

But during the startup it comes up with an exception of

Exception in thread "main" java.lang.IllegalStateException: Missing required properties: clientUrl
    at org.apache.beam.sdk.io.pulsar.AutoValue_PulsarIO_Write$Builder.build(AutoValue_PulsarIO_Write.java:84)
    at org.apache.beam.sdk.io.pulsar.PulsarIO.write(PulsarIO.java:148)
    at de.spx.bucketdataflow.PulsarTarget.main(PubSubToGcs.java:127)

After debugging, I come to the AutoValue_PulsarIO_Write$Builder.build() but there it is quiet simple.

        PulsarIO.Write build() {
            if (this.clientUrl == null) {
                String missing = " clientUrl";
                throw new IllegalStateException("Missing required properties:" + missing);
            } else {
                return new AutoValue_PulsarIO_Write(this.topic, this.clientUrl);
            }
        }

The check goes to the clientUrl which normally would get set in the next step of the builder pattern of PulsarIO.write().withClientUrl

Or am I handling it wrong?

I've tried beam-sdks-java-io-pulsar version 2.47 and 2.53 Both the same.

Thanks for hinting out!

-Tom

Answer 1

Make sure that the pulsarClientUrl variable enclosed in your .withClientUrl() has the correct and valid Pulsar broker URL as it is a required component in the PulsarIO.write() class and it is the one triggering the "Missing required properties: clientUrl" error message.

The AutoValue_PulsarIO_Write is a necessary class configuration that writes data to the Pulsar topic through the Pulsar IO connector. Its property this.clientUrl feeds on the same variable pulsarClientUrl.

Not sure if he is the same Tom as you but submitting a bug in GitHub (#30152) is a good start in seeking assistance to resolve this and hopefully be included on the next release.