Label Encoding for Categorical Features: Preserving Label Consistency Across Runs

Question

Problem Description:

Label Encoding Issue: Upon rerunning the label encoding code, the labels change, causing inconsistency.

Dynamic Data from a Server: Incoming data might introduce new values, making it impractical to predefine label limits.

Need for Persistent Labeling: Existing labels should remain consistent, while new values should get newly generated labels without altering existing labels.

Repetitive Function Runs: The code needs to handle multiple function runs.

Persistent Memory Between Program Runs: The program might restart, and the memory should retain label mappings to avoid rerunning from the start.

Existing Code:

from sklearn.preprocessing import LabelEncoder
import pickle

def label_encoding(df_logs):
    # Existing label mapping or an empty one
    try:
        with open('label_mapping.pkl', 'rb') as f:
            label_mapping = pickle.load(f)
    except FileNotFoundError:
        label_mapping = {}

    # Columns for label encoding
    cols_to_encode = [
        'Attack', 'Category', 'DstLocation', 'Os', 'SignName', 'SrcLocation', 'Target',
        'UserName', 'VSys', 'slot', 'Action', 'Policy', 'Profile', 'Protocol-Name',
        'Application', 'Source-zone', 'CloseReason', 'Destination-zone', 'ModuleName',
        'ModuleBrief', 'RecieveInterface', 'Policy-name', 'IP-address', 'Source-address', 'Destination-address'
    ]

    # Transform specific values in columns
    df_logs['Source-address'] = df_logs['Source-address'].apply(lambda x: '0' if x.startswith('192.168') else x)
    df_logs['Destination-address'] = df_logs['Destination-address'].apply(lambda x: '0' if x.startswith('192.168') else x)

    # Apply LabelEncoder to columns, maintain consistent labels
    for col in cols_to_encode:
        label_encoder = label_mapping.get(col, LabelEncoder())
        df_logs[col] = label_encoder.fit_transform(df_logs[col])
        label_mapping[col] = label_encoder  # Update label mappings

    # Save label mappings for future use
    with open('label_mapping.pkl', 'wb') as f:
        pickle.dump(label_mapping, f)

    return df_logs

Request:

Seeking a solution to maintain consistent labels for existing values across multiple runs while allowing newly encountered values to receive new labels without disrupting existing mappings. The goal is to preserve these mappings between program executions even after system restarts. Looking for suggestions or approaches to achieve this persistence and consistency in label encoding.