Generate Ethereum addresses in HD Wallet using public key only (bitcoinj/web3j)

Question

I trying to generate Ethereum addresses for the HD Wallet keys implemented with bitcoinj library, but I got confused:

DeterministicSeed seed = new DeterministicSeed("some seed code here", null, "", 1409478661L);
DeterministicKeyChain chain = DeterministicKeyChain.builder().seed(seed).build();
DeterministicKey addrKey = chain.getKeyByPath(HDUtils.parsePath("M/44H/60H/0H/0/0"), true);
System.out.println("address from pub=" + Keys.getAddress(Sign.publicKeyFromPrivate(addrKey.getPrivKey())));

this code prints a correct Ethereum address accordingly to https://iancoleman.io/bip39/. Everything is fine here.

But when I trying to avoid private key usage and generate non-hardened keys using public keys only I getting different results, i.e. the call returns another result:

System.out.println("address from pub=" + Keys.getAddress(addrKey.getPublicKeyAsHex()));

And it looks like the issue is in the "different public keys", i.e. result of the Sign.publicKeyFromPrivate(addrKey.getPrivKey()) and addrKey.getPublicKeyAsHex() are different. I'm not experienced with cryptography, thus it may be a silly question... but I would appreciate any advice here.

Answer 1

Like Bitcoin, Ethereum uses secp256k1. Ethereum addresses are derived as follows:

• Step 1: The 32 bytes x and y coordinate of the public key are concatenated to 64 bytes (where both the x and y coordinate are padded with leading 0x00 values if necessary).
• Step 2: From this the Keccak-256 hash is generated.
• Step 3: The last 20 bytes are used as the Ethereum address.

For the examples used here, the key is generated with:

String mnemonic = "elevator dinosaur switch you armor vote black syrup fork onion nurse illegal trim rocket combine";
DeterministicSeed seed = new DeterministicSeed(mnemonic, null, "", 1409478661L);
DeterministicKeyChain chain = DeterministicKeyChain.builder().seed(seed).build();
DeterministicKey addrKey = chain.getKeyByPath(HDUtils.parsePath("M/44H/60H/0H/0/0"), true);

This corresponds to the following public key and Ethereum address:

      X: a35bf0fdf5df296cc3600422c3c8af480edb766ff6231521a517eb822dff52cd
      Y: 5440f87f5689c2929542e75e739ff30cd1e8cb0ef0beb77380d02cd7904978ca
Address: 23ad59cc6afff2e508772f69d22b19ffebf579e7

as can also be verified with the website https://iancoleman.io/bip39/.

---

Step 1:

In the posted question, the expressions Sign.publicKeyFromPrivate() and addrKey.getPublicKeyAsHex() provide different results. Both functions return the public key in different types. While Sign.publicKeyFromPrivate() uses a BigInteger, addrKey.getPublicKeyAsHex() provides a hex string. For a direct comparison, BigInteger can be converted to a hex string with toString(16). When the results of both expressions are displayed with:

System.out.println(Sign.publicKeyFromPrivate(addrKey.getPrivKey()).toString(16));
System.out.println(addrKey.getPublicKeyAsHex());

the following result is obtained:

a35bf0fdf5df296cc3600422c3c8af480edb766ff6231521a517eb822dff52cd5440f87f5689c2929542e75e739ff30cd1e8cb0ef0beb77380d02cd7904978ca
02a35bf0fdf5df296cc3600422c3c8af480edb766ff6231521a517eb822dff52cd

The output of Sign.publicKeyFromPrivate() has a length of 64 bytes and corresponds to the concatenated x and y coordinate as defined in step 1. Therefore, the address generated with this is a valid Ethereum address, as also described in the posted question.

The output of addrKey.getPublicKeyAsHex(), on the other hand, corresponds to the x coordinate prefixed with a 0x02 value. This is the compressed format of the public key. The leading byte has either the value 0x02 if the y value is even (as in this example), or the value 0x03. Since the compressed format does not contain the y coordinate, this cannot be used to directly infer the Ethereum address, or if it is done anyway, it will result in a wrong address (indirectly, of course, it would be possible since the y coordinate can be derived from a compressed public key).

---

The uncompressed format of the public key can be obtained, e.g. with addrKey.decompress():

System.out.println(addrKey.decompress().getPublicKeyAsHex());

which gives this result:

04a35bf0fdf5df296cc3600422c3c8af480edb766ff6231521a517eb822dff52cd5440f87f5689c2929542e75e739ff30cd1e8cb0ef0beb77380d02cd7904978ca

The uncompressed format consists of a leading marker byte with the value 0x04 followed by the x and y coordinates. So if the leading marker byte is removed, just the data according to step 1 is obtained, which is needed for the derivation of the Ethereum address:

System.out.println(addrKey.decompress().getPublicKeyAsHex().substring(2));  

which results in:

a35bf0fdf5df296cc3600422c3c8af480edb766ff6231521a517eb822dff52cd5440f87f5689c2929542e75e739ff30cd1e8cb0ef0beb77380d02cd7904978ca

---

Steps 2 and 3:

Steps 2 and 3 are performed by Keys.getAddress(). This allows the Ethereum address to be obtained using the uncompressed public key as follows:

System.out.println(Keys.getAddress(addrKey.decompress().getPublicKeyAsHex().substring(2)));
System.out.println(Keys.getAddress(Sign.publicKeyFromPrivate(addrKey.getPrivKey())));       // For comparison

which gives the Ethereum address:

23ad59cc6afff2e508772f69d22b19ffebf579e7
23ad59cc6afff2e508772f69d22b19ffebf579e7

---

Overloads of Keys.getAddress():

Keys.getAddress() provides various overloads for the data types BigInteger, hex string and byte[]. If the uncompressed key is given as byte[], e.g. with addrKey.getPubKeyPoint().getEncoded(false), the byte[] can be used directly after removing the marker byte. Alternatively, the byte[] can be converted to a BigInteger with the marker byte removed:

byte[] uncompressed = addrKey.getPubKeyPoint().getEncoded(false);
System.out.println(bytesToHex(Keys.getAddress(Arrays.copyOfRange(uncompressed, 1, uncompressed.length))).toLowerCase());  // bytesToHex() from https://stackoverflow.com/a/9855338
System.out.println(Keys.getAddress(new BigInteger(1, uncompressed, 1, uncompressed.length - 1)));
System.out.println(Keys.getAddress(Sign.publicKeyFromPrivate(addrKey.getPrivKey())));                                     // For comparison

which as expected returns the same Ethereum address:

23ad59cc6afff2e508772f69d22b19ffebf579e7
23ad59cc6afff2e508772f69d22b19ffebf579e7
23ad59cc6afff2e508772f69d22b19ffebf579e7

One thing to note here is that Keys.getAddress(byte[]) does not pad the passed byte[], while the overloads for BigInteger or hex strings implicitly pad. This can be relevant e.g. when converting a BigInteger (e.g. provided by Sign.publicKeyFromPrivate(addrKey.getPrivKey())) to a byte[], since the result can also have less than 64 bytes (which would lead to different Keccak-256 hashes). If Keys.getAddress(byte[]) is used in this case, it must be explicitly padded with leading 0x00 values up to a length of 64 bytes.