Hexadecimal format payload decoding

Question

I have an home assignment where I need to decode hexadecimal payload. I have been trying to decode the payload but not able to produce correct output specifically for the time field. I would appreciate assistance in decoding it.

Description about the task:

In order to monitor the health of the battery, we decided to send data from the battery to the cloud. This data is transmitted in hexadecimal format and received in our AWS account.

The data is transmitted as a hexadecimal string. Every payload consists of 8 bytes. Due to space optimization, the information is not byte-aligned. A field can start in the middle of a byte. We therefore need bit operations to decode the payload. The payload is not signed and encoded in little Endian. The following table describes the data fields contained in the payload and their bit positions.

For instance, type is encoded on 4 bits in the first byte. state of charge is encoded on 8 bits (1 byte) on the 6th byte.

Time: time represents the timestamp of the data. It is defined in seconds since UNIX epoch.

State: state is a string, with the following corresponding values:

0: "power off"

1: "power on"

2: "discharge"

3: "charge"

4: "charge complete"

5: "host mode"

6: "shutdown"

7: "error"

8: "undefined"

State of charge: state of charge represents the charge of the battery. It is a float with values between 0 and 100 and a 0.5 precision. To store it as an integer, it was multiplied by 2.

Battery temperature: battery temperature represents the temperature of the battery. Values can vary between -20 and 100. The precision is 0.5. To store it as an integer we added 20 and multiplied it by 2.

Sample Test data

input: F1E6E63676C75000

output: { "time": 1668181615, "state": "error", "state_of_charge": 99.5, "temperature": 20.0 }

My Script:

import base64
import struct
import json
from datetime import datetime


def lambda_handler(event):
    # Extract device and payload from the event
    device = event["device"]
    payload_hex = event["payload"]

    # Convert hexadecimal payload to bytes
    payload_bytes = bytes.fromhex(payload_hex)

    # Unpack the payload using struct
    unpacked_data = struct.unpack('<I2Bh', payload_bytes)

    # Extract individual fields
    time = unpacked_data[0]

    state = unpacked_data[1]
    state = (state >> 4) & 0x0F

    state_of_charge = unpacked_data[2] / 2.0
    temperature = (unpacked_data[3] / 2.0) - 20.0

    # Mapping state values to corresponding strings
    state_mapping = {
    0: "power off",
    1: "power on",
    2: "discharge",
    3: "charge",
    4: "charge complete",
    5: "host mode",
    6: "shutdown",
    7: "error",
    8: "undefined"
    }

    # Create the output dictionary
    output_data = {
    "device": device,
    "time": time,
    "state": state_mapping.get(state, "unknown"),
    "state_of_charge": round(state_of_charge, 1),
    "temperature": round(temperature, 1)
    }

    # Log the output data to stdout
    print(json.dumps(output_data))

event = {'device': 'device1', 'payload': '6188293726C75C00'}
lambda_handler(event)

I am struggling to get the correct output for the time which is not just dependent on the unpacked_data[0] based on the above logic.

Answer 1

The struct library is not particularly useful since the data is not byte-aligned, so you'll need to do some manual bit-work. In case you haven't seen the word before, half of a byte is called a nibble; this program requires splitting bytes into nibbles.

# parse the hexadecimal payload
payload = bytes.fromhex("6188293726C75C00")

# split the payload into individual bytes
b0, b1, b2, b3, b4, b5, b6, b7 = payload

# extact the type from the least significant 4 bits on the first byte
type = b0 & 0x0F

# extract the time from the next 32 bits
time_nibbles = [
    b0 >> 4, # bits 0 - 4
    b1 & 0x0F, b1 >> 4, # bits 5 - 12
    b2 & 0x0F, b2 >> 4, # bits 13 - 20
    b3 & 0x0F, b3 >> 4, # bits 21 - 28
    b4 & 0x0F, # bits 29 - 32
]
time_bytes = [low | (high << 4) for low, high in zip(time_nibbles[::2], time_nibbles[1::2])]
time = int.from_bytes(time_bytes, byteorder="little")
print(f"{time = }")

# extract state
state = b4 >> 4
print(f"{state = }")

# extract state of charge
state_of_charge = b5 / 2
print(f"{state_of_charge = }")

# extract battery temperature
battery_temperature = (int.from_bytes([b6, b7], byteorder="little") / 2) - 20
print(f"{battery_temperature = }")

This outputs:

time = 1668454534
state = 2
state_of_charge = 99.5
battery_temperature = 26.0

Answer 2

You can use ctypes structures which support bitfields. Add properties to the class to do the calculations for you too:

import ctypes as ct
import struct

class _Data(ct.Structure):
    # Bitfield definitions
    _fields_ = (('type', ct.c_uint64, 4),
                ('time', ct.c_uint64, 32),
                ('state', ct.c_uint64, 4),
                ('soc', ct.c_uint64, 8),
                ('temp', ct.c_uint64, 8))

    states = ('power off', 'power on', 'discharge', 'charge',
              'charge complete', 'host mode', 'shutdown',
              'error', 'undefined')

class Data(ct.Union):
    _fields_ = (('_u', ct.c_uint64),
                ('_d', _Data))

    def __init__(self, data):
        # Take raw byte data and unpack as 64-bit little-endian
        self._u = struct.unpack('<Q', data)[0]

    @property
    def type(self):
        return self._d.type

    @property
    def time(self):
        return self._d.time

    @property
    def state(self):
        try:
            return self._d.states[self._d.state]
        except IndexError:
            return 'invalid'

    @property
    def state_of_charge(self):
        return self._d.soc / 2

    @property
    def battery_temperature(self):
        return self._d.temp / 2 - 20

    def __repr__(self):
        return (f'Data(type={self.type}, '
                     f'time={self.time}, '
                     f'state={self.state!r}, '
                     f'state_of_charge={self.state_of_charge}, '
                     f'temperature={self.battery_temperature})')

    def as_dict(self):
        return {'type': self.type,
                'time': self.time,
                'state': self.state,
                'state_of_charge': self.state_of_charge,
                'temperature': self.battery_temperature}

data = Data(bytes.fromhex('F1E6E63676C75000'))
print(data)
print(data.as_dict())

Output:

Data(type=1, time=1668181615, state='error', state_of_charge=99.5, temperature=20.0)
{'type': 1, 'time': 1668181615, 'state': 'error', 'state_of_charge': 99.5, 'temperature': 20.0}

You can also unpack the 64-bit value and extract bits via shifting and masking:

import struct

states = ('power off', 'power on', 'discharge', 'charge',
          'charge complete', 'host mode', 'shutdown',
          'error', 'undefined')

def take_bits(value, n):
    '''Remove n LSBs and shift value left n bits.'''
    return value >> n, value & (1 << n) - 1

def decode(data):
    value = struct.unpack('<Q', data)[0]
    # value = int.from_bytes(data, 'little') # another option
    value, type_ = take_bits(value, 4)
    value, time = take_bits(value, 32)
    value, state = take_bits(value, 4)
    value, charge = take_bits(value, 8)
    value, temp = take_bits(value, 8)
    return type_, time, state, charge / 2, temp / 2 - 20

print(decode(bytes.fromhex('F1E6E63676C75000')))
print(decode(bytes.fromhex('6188293726C75C00')))

Output:

(1, 1668181615, 'error', 99.5, 20.0)
(1, 1668454534, 'discharge', 99.5, 26.0)