How to configure SVM in the last layer of CNN in tensorflow?

Question

I came across this paper on "Deep Learning using Support Vector Machines". After watching this youtube tutorial, I tried to implement it in my model which is being trained on FER2013 dataset for facial human expression recognition.

Model:
Kaggle notebook

model = keras.Sequential([
    
    layers.Reshape((48, 48, 1), input_shape=(2304,)),
    
    layers.BatchNormalization(),
    layers.Conv2D(filters=64, kernel_size=3, activation='relu' ),
    layers.AveragePooling2D(pool_size=(2, 2)),
    layers.Dropout(0.5),
    
    layers.BatchNormalization(),
    layers.Conv2D(filters=128, kernel_size=3, activation='relu'),
    layers.AveragePooling2D(pool_size=(2, 2)),
    layers.Dropout(0.5),
    
    layers.BatchNormalization(),
    layers.Conv2D(filters=128, kernel_size=3, activation='relu'),
    layers.AveragePooling2D(pool_size=(2, 2)),
    layers.Dropout(0.5),
    
    layers.BatchNormalization(),
    layers.Conv2D(filters=512, kernel_size=3, activation='relu'),
    layers.AveragePooling2D(pool_size=(2, 2)),
    layers.Dropout(0.5),
    
    layers.Flatten(),
    
    layers.BatchNormalization(),
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.3),
    
    layers.BatchNormalization(),
    layers.Dense(256, activation='relu'),
    layers.Dropout(0.3),
    
    layers.Dense(7, kernel_regularizer=tf.keras.regularizers.l2(0.01),activation
             ='softmax')
])

model.compile(
    optimizer='adam',
    loss = 'squared_hinge',
    metrics=['accuracy'],
)

But this is giving unexpected results:

Epoch 47/50
202/202 [==============================] - 4s 19ms/step - loss: 0.3469 - accuracy: 0.1308 - val_loss: 0.3508 - val_accuracy: 0.1299
Epoch 48/50
202/202 [==============================] - 4s 19ms/step - loss: 0.3469 - accuracy: 0.1326 - val_loss: 0.3508 - val_accuracy: 0.1378
Epoch 49/50
202/202 [==============================] - 4s 19ms/step - loss: 0.3469 - accuracy: 0.1617 - val_loss: 0.3508 - val_accuracy: 0.2426
Epoch 50/50
202/202 [==============================] - 4s 19ms/step - loss: 0.3469 - accuracy: 0.1594 - val_loss: 0.3508 - val_accuracy: 0.1291

What am I doing wrong? How to fix this?