Understanding DeepAr plot_prediction in pytorch forecasting

Question

I have tried the example of the pytorch forecasting DeepAR implementation as described in the doc.

There are two ways to create and plot predictions with the model, which give very different results. One is using the model's forward() function and the other the model's predict() function.

One way is implemented in the model's validation_step() method and is logged in tensorboard:

# validation step
idx = 0

prediction_kwargs = {}
prediction_kwargs.setdefault("n_samples", 20)
prediction_kwargs.setdefault("use_metric", True)

quantiles_kwargs = {}
quantiles_kwargs.setdefault("n_samples", 20)
quantiles_kwargs.setdefault("use_metric", True)
  
kwargs = dict(prediction_kwargs=prediction_kwargs, quantiles_kwargs=quantiles_kwargs)

x, y = next(iter(val_dataloader))
out = net(x, n_samples=None)
fig = net.plot_prediction(x, out, idx=idx, add_loss_to_title=True, **kwargs)

After 5 epochs

The other way is described in the doc:

# doc
idx = 0

raw_prediction, x = net.predict(
    validation,
    mode="raw", return_x=True)

import matplotlib.pyplot as plt
fig = net.plot_prediction(x, raw_prediction, idx=idx, add_loss_to_title=True)

After 5 epochs

I am using pytorch=1.13.1, pytorch_lightning=1.8.6 and pytorch_forecasting=0.10.2.

Thanks for an input.

Answer 1

The difference comes from the model's parameter n_samples, which is explicitly set to None in the first case, while it is implicitly set to 100 in the second case.

According to the code comment "If n_smaples [sic] is given, decode not by using actual values but rather by sampling new targets from past predictions iteratively". Hence with n_samples=None, the next time prediction is based on the actual value, and not the predicted value, of the current time.