Add fit lines from an nlme model to a ggplot, and include confidence intervals

Question

I've used nlme to model my data like this:

library(nlme)
my_model <- lme(metric ~ group * time, random = "~1 | tech_rep", na.action = "na.omit", data = my_data)

I then want to plot my data, but include fit lines from the nlme model. I gather from some posts that this can be done with ggpredict (Extract prediction band from lme fit). I've managed to add the lines, but I'd also like the confidence interval to be displayed as a shared area around the line (as is default for geom_smooth). Setting se = TRUE doesn't seem to do anything.

ggpredict_data <- ggpredict(my_model, c("time", "group"), type = "re")
ggplot(ggpredict_data, aes(x = x, y = predicted, colour = group, fill = group)) +
    stat_smooth(method = "lm", se = TRUE, fullrange = TRUE) +
    geom_point(data = my_data,
               aes(x = time, y = metric, colour = group))

Can anyone help get the confidence interval working?

R, reading around, trying other posts