Unable to extract features using huggingface swin transformer model

Question

I need to use swin transfomer as a backbone for my SVM model. I need to extract the features first, therefore I am trying to remove the last layer and train the model. This is a sample executable code.

import torch
import torch.nn as nn
import torch.optim as optim
import torchvision
import torchvision.transforms as transforms
import numpy as np

HUB_URL = "SharanSMenon/swin-transformer-hub:main"
MODEL_NAME = "swin_tiny_patch4_window7_224"
model = torch.hub.load(HUB_URL, MODEL_NAME, pretrained=True)
model = nn.Sequential(*list(model.children())[:-1])          #--> If you comment this line, the code is executing
for param in model.parameters():
    param.requires_grad = False


dummy_tensor = torch.randn(32, 3, 224, width)

# Print the shape of the dummy tensor
model(dummy_tensor)

What am I missing, I tried the same process with resnet50 and it worked fine. Why am I not able to remove the last layer and get only the output features? what should I do? any suggestions?

Answer 1

The two models have different forward logic. If you inspect the result of nn.Sequential(*list(model.children())[:-1]), you'll see you're putting a ModuleList into a Sequential block which doesn't make sense. The specific error is trying to call the forward method of ModuleList which isn't implemented.

You need to adapt your logic to each model architecture.