WPF ListView initialization slow - can't get virtualization to work

Question

I have a ListView that I can only initialize in my UserControl's constructor. There, I get from my database an IEnumerable of around 500 items. (I've tested, and this operation alone is quite fast.)

However, once I set my ListView.ItemsSource to my IEnumerable, it takes around 5-10 seconds to show all entries in the ListView.

I've read about virtualization, and it seems to be exactly what I need. So I defined my ListView as follows:

<ListView
    VirtualizingStackPanel.IsVirtualizing="True"
    ItemsSource="{Binding AllComponents}"
    SelectionMode="Extended">

    <ListView.View>
        <GridView>
            <GridViewColumn Header="Name">
                <GridViewColumn.CellTemplate>
                    <DataTemplate>
                        <TextBlock Text="{Binding Component.name}"/>
                    </DataTemplate>
                </GridViewColumn.CellTemplate>
            </GridViewColumn>
        </GridView>
    </ListView.View>
</ListView>

(I also have some other columns, but removed them for this example. Even with just Name it is quite slow.)

In my code-behind, in my UserControl's constructor, I initialize a view model to hold AllComponents, and then set the DataContext to that view model.

It still takes the same time to populate the list as if there's no virtualization at all (if I set the binding to IsAsync=True, I can time how long it takes for the list to show). Additionally, if I retrieve all 500 items, and then give my listview the first 20, it is much faster. So it seems like there's no virtualization at all. (Otherwise, I would expect a virtualized 500 items to take around the same time as non-virtualized 20 items.)

So I think I'm doing something wrong or not setting up the virtualization properly. I would appreciate any help!

Edit: As far as I can tell, I don't have a custom style either (I've seen that can be a problem for virtualization).

Another edit: Apparently, using MahApps.Metro does create a custom style. Marked this post as a duplicate since it is basically the same question. The answer there suggested to add the following to the ListView:

<ListView
    ScrollViewer.CanContentScroll="True"
    ScrollViewer.IsDeferredScrollingEnabled="True" 
    VirtualizingPanel.IsVirtualizing="True"
    VirtualizingPanel.IsVirtualizingWhenGrouping="True" .../>