Getting timeout while creating ACL for Amazon Kafka

Question

I'm setting up a public kafka cluster with SCRAM auth on AWS MSK. I created the cluster, attached a secret from KMS and made it public.

When I'm trying to connect it with kcat, I'm getting topic authorization error.

$ echo "message" | kcat -L -b "broker1:port,broker2:port" -t "test-topic" -p 1 -X security.protocol=SASL_SSL -X sasl.mechanism="SCRAM-SHA-512" -X sasl.username=<user-from-kms> -X sasl.password=<password-from-kms>

Delivery failed for message: Broker: Topic authorization failed

So I'm trying to create the ACLs reading this using the kafka docker container, but getting timeout error in node assignment.

$ /bin/kafka-acls --bootstrap-server "broker:port"  --command-config config.properties --add --allow-principal User:<kms-user> --operation Read --topic test_topic

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=superbio_ui_events_staging, patternType=LITERAL)`:
    (principal=User:gossupkafkaprod, host=*, operation=READ, permissionType=ALLOW)

Error while executing ACL command: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createAcls
...

Content of config.properties

$ cat config.properties
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
        username="<kms-user>" \
        password="<kms-password>";
ssl.truststore.location=./kafka.client.truststore.jks

I've disabled the IAM auth as well, only the SASL/SCRAM is enabled with SSL in transit. I also had to allow.everyone.if.no.acl.found to false to make the cluster public.

I tried following this, i.e. create topic with zookeepers where I'm not sure how to provide scram credentials but that's also giving timeout error.

$ /bin/kafka-acls --authorizer-properties zookeeper.connect="broker:port" --add --allow-principal "User:<kms-user>" --operation Write --topic

...
Client session timed out, have not heard from server
...

Answer 1

I think you almost there!

1. Since you enabled public access before setting up ACLs, it makes sense you are getting authorization error.

2. Your assumption to use Zookeeper to set ACLs is correct. Moreover, your command is perfectly fine, you don't need to provide any authentication to ZK, because it doesn't support, it uses either plaintext or TLS encryption.

I'd suggest you to:

• Check security groups inbound rules that they allow access to ports 2181 (paintext) or 2182 (TLS)
• since ZK don't support authentication, I'd recommend to have those rules opened to SG/resources in your VPC only

After that you supposed to have a connectivity.

For further connectivity troubleshooting, you can use telnet <ZK-host> 2181. If you have a prompt and no timeout, you should be good to run ACLs commands against ZK.