Failed to Find Any Kerberos TGT while trying to access Kerberized HBase Without kinit

Question

I have a very simple Scala HBase GET application. I tried to make the connection as below:

import org.apache.hadoop.hbase.{HBaseConfiguration, TableName}
import org.apache.hadoop.hbase.client.{ConnectionFactory, Get}

object Debug extends App {
    val hbaseConf: HadoopConf = HBaseConfiguration.create
    val connection: Connection = ConnectionFactory.createConnection(hbaseConf)
    val hbaseTable = connection.getTable(TableName.valueOf("my-hbase-table"))
    hbaseTable.get(new Get("rowkey".getBytes).addColumn("colFam".getBytes,"colName".getBytes))
}

Whenever I run this, I get an error like below:

Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.hbase.security.AbstractHBaseSaslRpcClient.getInitialResponse(AbstractHBaseSaslRpcClient.java:131)
at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:108)
at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:104)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.handlerAdded(NettyHBaseSaslRpcClientHandler.java:104)
at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:606)
at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:187)
at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:380)
at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:359)
at org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnection.java:200)
... 18 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:162)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:189)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 30 more

I am using Windows, so I have put 4 files under the root directory of C: drive:

    C:\cacerts
    C:\jaas.conf
    C:\krb5.conf
    C:\principal.keytab

My C:\jaas.conf :

KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="C:\\principal.keytab"
    useTicketCache=false
    debug=true
    principal="principal@REALM";
};
Client {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="C:\\principal.keytab"
    useTicketCache=false
    debug=true
    principal="principal@REALM";
};
com.sun.security.jgss.krb5.initiate {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    keyTab="C:\\principal.keytab"
    principal="principal@REALM";
};

I am on Cloudera CDH version 6.3.2. Downloaded hbase-client-config (ssl-client.xml, hdfs-site.xml, hbase-site.xml, core-site.xml) from Cloudera manager and added HBase client config files under resources folder in IntelliJ. my-hbase-client-config-files

And also set VM options for jaas.conf in IntelliJ, Run/Debug Configurations -> Application -> Build and run -> VM Options. my-intellij-application-vm-options-config

When application is started, am able to see jaas.conf file is being taken jaas.conf-logs

Even if it is saying that it gonna use the keytab, still I am getting error like this: Kerberos-error-message

My Cloudera version: 6.3.2

Scala version: 2.11.12

HBase Client: 2.1.0

Does anyone has any ideas? Is Java authentication system not taking Kerberos ticket by itself when I give jaas.conf?

Answer 1

I'm not sure if HBase client implements Service Provider Interface (which would enable the use of JAAS-style security config).

Perhaps you can try to add UGI initialization in your Scala code directly, like below?

object Debug extends App {
   val hbaseConf: HadoopConf = HBaseConfiguration.create

   UserGroupInformation.setConfiguration(hbaseConf)
   UserGroupInformation.loginUserFromKeytab("principal@REALM","C:\\principal.keytab")

   val connection: Connection = ConnectionFactory.createConnection(hbaseConf)
   :
   etc
   :

Otherwise, could you share a reference/link to any information that suggests that it is configurable thru jaas.conf?

Answer 2

You will get this error message when Jaas cannot access the kerberos keytab.

Can you check for user permission issues? Login as user that will run the code and do a kinit ? What error message do you get? (Resolve the permission issue I'm suggesting you have.)

You seem to rule out a path issue, and seem to have the correct '\\'.