GCP PubSub: Publishing messages takes time when there is a spike in incoming messages to publish

69 Views Asked by At

Setup:

  1. Springboot & Kotlin app hosted in GCP GKE
  2. Above app receives data from AMQP/rabbitMQ service
  3. Above app receives AMQP data and publishes into Google cloud PubSub , GCP GKE and PubSub are in different GCP projects , data published using service account
  4. PubSub Push subscription - Cloud function . Memory 512MB, FirstGen, Maximum 3000 instances

Problem: Sometimes when there is a spike in AMQP messages seeing delays in publish around 20-60 seconds. For instance after receiving the message from AMQP , time taken to publish is around 20-60 seconds.

Questions:

  1. Is my way of logging correct, does publish time means publishing to push subscription or invoked cloud function completion.
  2. What do i need to do to increase the throughput? These messages are time sensitive means they need to be delivered as soon as its created.
  3. Is there something i need to consider to scale when there is a spike . like horizontal scaling , multiple publish clients ,etc.,
fun publishCode(receivedTime:LocalDateTime){
    PubsubMessage.newBuilder()
            .setData(ByteString.copyFrom(message.body))
            .build()?.let {
                val apiFuture: ApiFuture<String> = publisher.publish(it) //publisher created using bean
                ApiFutures.addCallback(apiFuture, object : ApiFutureCallback<String> {
                    override fun onSuccess(messageID: String?) {
                        val publishedTime = getCurrentTime()
                        log.info("Published success : Received:$receivedTime Published:$publishedTime"
                    }
                })
            } 
}

@Bean
fun publisher(): Publisher {
    return Publisher.newBuilder("$topicToPublish").setRetrySettings(retrySettings)
            .build()
}

0

There are 0 best solutions below