How to run python kafka application with multiple kerberos caches?

Question

I have a python microservice application that on request sends specific message to specific kafka topic with specific principal. All parameters are user controlled in request body. How do I avoid Kafka thread safety problems?

I do know that I can control the kerberos cache via env variable KRB5CCNAME, but if I touch it, it won't be thread safe (one thread of Kafka might collide with another one afterall not letting me reach right topic with right credentials). If I do it via multiprocessing then it won't be memory efficient as I need to deal with potentially hundred of such topics. If I don't touch KRB5CCNAME then each kinit command will overwrite same cache with new ticket this way leading me to security leak (wrong app can publish to wrong topic).

I'm not in control of infrastructure and cannot opt into changing design pattern of architecture (can't get one principal with access to many topics, it has to be "bring your own credentials and topic" approach)

How to achieve concurrent Kafka sessions with different principals simultaneously from single python process without multiprocessing?

I'm using confluent-kafka-python (and accordingly librdkafka) with vanilla Flask on RHEL8 with MIT kerberos.

Answer 1

Instead of letting libgssapi find credentials on its own, parts of the librdkafka library would need to be rewritten to explicitly acquire Kerberos credentials using the GSSAPI Credential Store extensions now available in MIT libkrb5 (and I believe in Heimdal as well).

The gssproxy daemon and the Apache mod_auth_gssapi module are good examples of using these extensions in C code to access multiple sets of credentials within the same process. For pure Python code, python-gssapi makes these extensions easy to use via gssapi.Credential – ideally the Python kafka bindings would use that somehow.

But other than actually rewriting the parts of the library that read process-global environment variables to Not Do That, there's no secret backdoor to make a non-threadsafe library somehow thread-safe. (Except for, perhaps, separating the Kafka-related code into its own daemon and ensuring that it would fork a new process for every topic – it shouldn't be horribly inefficient, as 'multiprocessing' appears to use fork() instead of spawning a whole new process, i.e. much of the memory is actually shared copy-on-write.)

If I don't touch KRB5CCNAME then each kinit command will overwrite same cache with new ticket this way leading me to security leak (wrong app can publish to wrong topic).

This is not necessarily true with current MIT Kerberos or Heimdal Kerberos. While the FILE:-based caches can only hold tickets for a single principal, both libraries also support DIR:<path> for directory-based cache collections, where each new kinit only adds a new ticket cache (and klist -A would show all). I think Heimdal's SQLITE: is multi-principal as well.

While DIR still has a concept of "active" cache for legacy APIs, a standard request for a service principal ticket will use all of them – not just the active one. With MIT Kerberos, the algorithm for choosing between caches in a collection can be influenced by ~/.k5identity to map service principal domains to Kerberos realms.

Unfortunately, this will be completely useless in your case, as it sounds like all credentials are used for the exact same server.