Is there a way to standardize color order in a bar plot while using scale_fill_viridis, or similar option?

Question

I have dashboard for a survey I am working on. The survey questions range in how many responses are available from 2 to 16 responses.

For clarity and readability I would like to keep the scale consistent across questions and the preference of by boss is that something like a virdis scale is used so its legible if it is in color or reprinted in black and white.

I want to ensure that the responses will always keep the same color options so position 1 in the graph will always be blue, position 2 green, position 3 purple, etc. (this is just an example it doesn't need to be a specific color the color just needs to be consistent). What occurs now is that the colors offered shift based on the number of possible responses available.

Below is a snippet of the styling code that I am utilizing for reference

an_inhome_graph<-ggplot(data = an_inhome_df, aes(x = Response, y = n,
                                           fill=Response,
                                           tooltip = paste
                                           (n, "Participants"))) +
  geom_bar_interactive(stat = "identity",show.legend = FALSE) +
  ggtitle(paste("Q16: Do animals come inside the place where you live?"))+
  scale_fill_viridis_d(direction = 1) +
  scale_x_discrete(labels = c("No", "Yes"))+
  labs(x = NULL, y = NULL) +
  theme(  # Some styling
    text = element_text(family = "serif", colour = "#5EB9D6", size=12),
    axis.text = element_text(family = "serif", colour = "#5EB9D6", 
                             size=12,angle=90),
    legend.position = NULL,
    plot.background = element_rect(fill="white",colour="white"),
    panel.background = element_rect(fill="white",colour="white"))

I have tried to change the direction of viridis, as well as changing start and begin with no luck