How to get the accuracy validation per epoch or step for the huggingface.transformers Trainer?

19 Views Asked by At

I'm studying the transformers model, I have code like this

# define the metrics

import numpy as np
from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score, classification_report

def compute_metrics(p):
    pred, labels = p
    pred = np.argmax(pred, axis=1)

    report = classification_report(labels, pred, digits=4)
    acc = accuracy_score(y_true=labels, y_pred=pred)
    rec = recall_score(y_true=labels, y_pred=pred, average='micro')
    prec = precision_score(y_true=labels, y_pred=pred, average='micro')
    f1 = f1_score(y_true=labels, y_pred=pred, average='micro')

    print("Classification Report:\n{}".format(report))
    return {"accuracy": acc, "precision": prec, "recall": rec, "f1": f1}

from transformers import TrainingArguments, Trainer
from transformers import AutoModelForSequenceClassification

# Get the base model and configuration
model = AutoModelForSequenceClassification.from_pretrained(
    model_chkpt,
    num_labels=7,
    id2label=id2label,
    label2id=label2id,
    ignore_mismatched_sizes=True
)

output_dir = "indonesian-emotion-distilbert-base-cased-finetuned" # name directory
training_args = TrainingArguments(
        output_dir=output_dir,
        evaluation_strategy="epoch",
        save_strategy="epoch",
        learning_rate=1e-5,
        per_device_train_batch_size=16,
        per_device_eval_batch_size=16,
        num_train_epochs=5,
        warmup_ratio=0.01,
        weight_decay=0.01,
        load_best_model_at_end=True,
        metric_for_best_model="accuracy",
        push_to_hub=True)

trainer = Trainer(
    model=model,
    args=training_args,
    tokenizer=tokenizer,
    train_dataset=tokenized_train_data,
    eval_dataset=tokenized_test_data,
    compute_metrics=compute_metrics,
)

trainer.train()

in the evaluation results I got results like this

Training Loss, Epoch Step Validation Loss Accuracy Precision Recall F1

Is the accuracy in this matrix validation or training?

If the accuracy matrix is ​​training, how do I get the validation accuracy matrix

0

There are 0 best solutions below