Plotting points using usmap and ggplot2

Question

I currently have to an excel datasheet of longitudes, latitudes, and a third category called "quarter" and here's how I'm plotting it:

plot_usmap("states")+
  labs(title="US Property Deletions by Quarter")+
  geom_point(data = long_lats,
             aes(x=lat, y=lon, color=quarter),
             size = 1)

This is my output:

The "quarter" breakdown appears exactly how I want, but none of the points are plotting from the datasheet. I'd appreciate any help, thank you.