Table Of Contents
Table Of Contents

[Download]

Quick Start Tutorial

The GluonTS toolkit contains components and tools for building time series models using MXNet. The models that are currently included are forecasting models but the components also support other time series use cases, such as classification or anomaly detection.

The toolkit is not intended as a forecasting solution for businesses or end users but it rather targets scientists and engineers who want to tweak algorithms or build and experiment with their own models.

GluonTS contains:

  • Components for building new models (likelihoods, feature processing pipelines, calendar features etc.)
  • Data loading and processing
  • A number of pre-built models
  • Plotting and evaluation facilities
  • Artificial and real datasets (only external datasets with blessed license)
In [1]:
# Third-party imports
%matplotlib inline
import mxnet as mx
from mxnet import gluon
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import json

Datasets

GluonTS datasets

GluonTS comes with a number of publicly available datasets.

In [2]:
from gluonts.dataset.repository.datasets import get_dataset, dataset_recipes
from gluonts.dataset.util import to_pandas
In [3]:
print(f"Available datasets: {list(dataset_recipes.keys())}")
Available datasets: ['constant', 'exchange_rate', 'solar-energy', 'electricity', 'traffic', 'm4_hourly', 'm4_daily', 'm4_weekly', 'm4_monthly', 'm4_quarterly', 'm4_yearly']

To download one of the built-in datasets, simply call get_dataset with one of the above names. GluonTS can re-use the saved dataset so that it does not need to be downloaded again: simply set regenerate=False.

In [4]:
dataset = get_dataset("m4_hourly", regenerate=True)
INFO:root:downloading and processing m4_hourly
saving time-series into /var/lib/jenkins/.mxnet/gluon-ts/datasets/m4_hourly/train/data.json
saving time-series into /var/lib/jenkins/.mxnet/gluon-ts/datasets/m4_hourly/test/data.json

In general, the datasets provided by GluonTS are objects that consists of three main members:

  • dataset.train is an iterable collection of data entries used for training. Each entry corresponds to one time series
  • dataset.test is an iterable collection of data entries used for inference. The test dataset is an extended version of the train dataset that contains a window in the end of each time series that was not seen during training. This window has length equal to the recommended prediction length.
  • dataset.metadata containts metadata of the dataset such as the frequency of the time series, a recommended prediction horizon, associated features, etc.
In [5]:
entry = next(iter(dataset.train))
train_series = to_pandas(entry)
train_series.plot()
plt.grid(which="both")
plt.legend(["train series"], loc="upper left")
plt.show()
../../_images/examples_basic_forecasting_tutorial_tutorial_8_0.png
In [6]:
entry = next(iter(dataset.test))
test_series = to_pandas(entry)
test_series.plot()
plt.axvline(train_series.index[-1], color='r') # end of train dataset
plt.grid(which="both")
plt.legend(["test series", "end of train series"], loc="upper left")
plt.show()
../../_images/examples_basic_forecasting_tutorial_tutorial_9_0.png
In [7]:
print(f"Length of forecasting window in test dataset: {len(test_series) - len(train_series)}")
print(f"Recommended prediction horizon: {dataset.metadata.prediction_length}")
print(f"Frequency of the time series: {dataset.metadata.freq}")
Length of forecasting window in test dataset: 48
Recommended prediction horizon: 48
Frequency of the time series: H

Custom datasets

At this point, it is important to emphasize that GluonTS does not require this specific format for a custom dataset that a user may have. The only requirements for a custom dataset are to be iterable and have a “target” and a “start” field. To make this more clear, assume the common case where a dataset is in the form of a numpy.array and the index of the time series in a pandas.Timestamp (possibly different for each time series):

In [8]:
N = 10  # number of time series
T = 100  # number of timesteps
prediction_length = 24
freq = "1H"
custom_dataset = np.random.normal(size=(N, T))
start = pd.Timestamp("01-01-2019", freq=freq)  # can be different for each time series

Now, you can split your dataset and bring it in a GluonTS appropriate format with just two lines of code:

In [9]:
from gluonts.dataset.common import ListDataset
In [10]:
# train dataset: cut the last window of length "prediction_length", add "target" and "start" fields
train_ds = ListDataset([{'target': x, 'start': start}
                        for x in custom_dataset[:, :-prediction_length]],
                       freq=freq)
# test dataset: use the whole dataset, add "target" and "start" fields
test_ds = ListDataset([{'target': x, 'start': start}
                       for x in custom_dataset],
                      freq=freq)

Training an existing model (Estimator)

GluonTS comes with a number of pre-built models. All the user needs to do is configure some hyperparameters. The existing models focus on (but are not limited to) probabilistic forecasting. Probabilistic forecasts are predictions in the form of a probability distribution, rather than simply a single point estimate.

We will begin with GulonTS’s pre-built feedforward neural network estimator, a simple but powerful forecasting model. We will use this model to demonstrate the process of training a model, producing forecasts, and evaluating the results.

GluonTS’s built-in feedforward neural network (SimpleFeedForwardEstimator) accepts an input window of length context_length and predicts the distribution of the values of the subsequent prediction_length values. In GluonTS parlance, the feedforward neural network model is an example of Estimator. In GluonTS, Estimator objects represent a forecasting model as well as details such as its coefficients, weights, etc.

In general, each estimator (pre-built or custom) is configured by a number of hyperparameters that can be either common (but not binding) among all estimators (e.g., the prediction_length) or specific for the particular estimator (e.g., number of layers for a neural network or the stride in a CNN).

Finally, each estimator is configured by a Trainer, which defines how the model will be trained i.e., the number of epochs, the learning rate, etc.

In [11]:
from gluonts.model.simple_feedforward import SimpleFeedForwardEstimator
from gluonts.trainer import Trainer
INFO:root:Using CPU
In [12]:
estimator = SimpleFeedForwardEstimator(
    num_hidden_dimensions=[10],
    prediction_length=dataset.metadata.prediction_length,
    context_length=100,
    freq=dataset.metadata.freq,
    trainer=Trainer(ctx="cpu",
                    epochs=5,
                    learning_rate=1e-3,
                    num_batches_per_epoch=100
                   )
)

After specifing our estimator with all the necessary hyperparameters we can train it using our training dataset dataset.train by invoking the train method of the estimator. The training algorithm returns a fitted model (or a Predictor in GluonTS parlance) that can be used to construct forecasts.

In [13]:
predictor = estimator.train(dataset.train)
INFO:root:Start model training
INFO:root:Number of parameters in SimpleFeedForwardTrainingNetwork: 483
INFO:root:Epoch[0] Learning rate is 0.001
100%|██████████| 100/100 [00:00<00:00, 100.76it/s, avg_epoch_loss=5.52]
INFO:root:Epoch[0] Elapsed time 0.995 seconds
INFO:root:Epoch[0] Evaluation metric 'epoch_loss'=5.518744
INFO:root:Epoch[1] Learning rate is 0.001
100%|██████████| 100/100 [00:00<00:00, 148.99it/s, avg_epoch_loss=4.82]
INFO:root:Epoch[1] Elapsed time 0.673 seconds
INFO:root:Epoch[1] Evaluation metric 'epoch_loss'=4.822126
INFO:root:Epoch[2] Learning rate is 0.001
100%|██████████| 100/100 [00:00<00:00, 186.65it/s, avg_epoch_loss=4.62]
INFO:root:Epoch[2] Elapsed time 0.537 seconds
INFO:root:Epoch[2] Evaluation metric 'epoch_loss'=4.620721
INFO:root:Epoch[3] Learning rate is 0.001
100%|██████████| 100/100 [00:00<00:00, 222.51it/s, avg_epoch_loss=4.72]
INFO:root:Epoch[3] Elapsed time 0.451 seconds
INFO:root:Epoch[3] Evaluation metric 'epoch_loss'=4.718864
INFO:root:Epoch[4] Learning rate is 0.001
100%|██████████| 100/100 [00:00<00:00, 194.80it/s, avg_epoch_loss=4.5]
INFO:root:Epoch[4] Elapsed time 0.515 seconds
INFO:root:Epoch[4] Evaluation metric 'epoch_loss'=4.499774
INFO:root:Loading parameters from best epoch (4)
INFO:root:Final loss: 4.499773736000061 (occurred at epoch 4)
INFO:root:End model training

With a predictor in hand, we can now predict the last window of the dataset.test and evaluate our model’s performance.

GluonTS comes with the make_evaluation_predictions function that automates the process of prediction and model evaluation. Roughly, this function performs the following steps:

  • Removes the final window of length prediction_length of the dataset.test that we want to predict
  • The estimator uses the remaining data to predict (in the form of sample paths) the “future” window that was just removed
  • The module outputs the forecast sample paths and the dataset.test (as python generator objects)
In [14]:
from gluonts.evaluation.backtest import make_evaluation_predictions
In [15]:
forecast_it, ts_it = make_evaluation_predictions(
    dataset=dataset.test,  # test dataset
    predictor=predictor,  # predictor
    num_eval_samples=100,  # number of sample paths we want for evaluation
)

First, we can convert these generators to lists to ease the subsequent computations.

In [16]:
forecasts = list(forecast_it)
tss = list(ts_it)

We can examine the first element of these lists (that corresponds to the first time series of the dataset). Let’s start with the list containing the time series, i.e., tss. We expect the first entry of tss to contain the (target of the) first time series of dataset.test.

In [17]:
# first entry of the time series list
ts_entry = tss[0]
In [18]:
# first 5 values of the time series (convert from pandas to numpy)
np.array(ts_entry[:5]).reshape(-1,)
Out[18]:
array([605., 586., 586., 559., 511.], dtype=float32)
In [19]:
# first entry of dataset.test
dataset_test_entry = next(iter(dataset.test))
In [20]:
# first 5 values
dataset_test_entry['target'][:5]
Out[20]:
array([605., 586., 586., 559., 511.], dtype=float32)

The entries in the forecast list are a bit more complex. They are objects that contain all the sample paths in the form of numpy.ndarray with dimension (num_samples, prediction_length), the start date of the forecast, the frequency of the time series, etc. We can access all these information by simply invoking the corresponding attribute of the forecast object.

In [21]:
# first entry of the forecast list
forecast_entry = forecasts[0]
In [22]:
print(f"Number of sample paths: {forecast_entry.num_samples}")
print(f"Dimension of samples: {forecast_entry.samples.shape}")
print(f"Start date of the forecast window: {forecast_entry.start_date}")
print(f"Frequency of the time series: {forecast_entry.freq}")
Number of sample paths: 100
Dimension of samples: (100, 48)
Start date of the forecast window: 1750-01-30 04:00:00
Frequency of the time series: H

We can also do calculations to summarize the sample paths, such computing the mean or a quantile for each of the 48 time steps in the forecast window.

In [23]:
print(f"Mean of the future window:\n {forecast_entry.mean}")
print(f"0.5-quantile (median) of the future window:\n {forecast_entry.quantile(0.5)}")
Mean of the future window:
 [632.3635  606.0122  503.5777  513.6542  450.2388  448.5289  452.94434
 497.69046 550.9761  548.6255  598.7145  670.1717  762.365   736.63416
 850.13617 883.0539  916.4168  827.62427 845.7739  867.36365 818.1774
 788.682   778.0424  729.4003  666.657   592.9861  539.96295 458.83582
 457.38782 486.90094 504.62613 456.41937 524.6526  514.2633  610.74475
 703.5968  739.01733 814.07263 900.44037 856.2595  850.2749  901.7657
 857.98474 852.56256 841.9727  761.44775 783.5231  711.0005 ]
0.5-quantile (median) of the future window:
 [637.573   605.9827  514.33307 514.0186  475.04318 450.99136 456.98206
 502.92288 564.31744 517.90094 598.58124 666.85345 776.13245 749.4729
 847.03296 881.6832  897.71783 834.5018  837.11365 860.62244 825.80585
 789.886   790.0141  729.37463 664.32544 586.74005 537.5214  467.923
 452.30966 476.7507  466.1437  447.1587  527.48346 513.9024  610.14734
 695.6569  733.7925  824.55804 894.59644 872.45776 857.4296  913.2191
 853.59576 846.80334 826.5576  759.4314  781.92206 705.6609 ]

Forecast objects have a plot method that can summarize the forecast paths as the mean, prediction intervals, etc. The prediction intervals are shaded in different colors as a “fan chart”.

In [24]:
def plot_prob_forecasts(ts_entry, forecast_entry):
    plot_length = 150
    prediction_intervals = (50.0, 90.0)
    legend = ["observations", "median prediction"] + [f"{k}% prediction interval" for k in prediction_intervals][::-1]

    fig, ax = plt.subplots(1, 1, figsize=(10, 7))
    ts_entry[-plot_length:].plot(ax=ax)  # plot the time series
    forecast_entry.plot(prediction_intervals=prediction_intervals, color='g')
    plt.grid(which="both")
    plt.legend(legend, loc="upper left")
    plt.show()
In [25]:
plot_prob_forecasts(ts_entry, forecast_entry)
../../_images/examples_basic_forecasting_tutorial_tutorial_38_0.png

We can also evaluate the quality of our forecasts numerically. In GluonTS, the Evaluator class can compute aggregate performance metrics, as well as metrics per time series (which can be useful for analyzing performance across heterogeneous time series).

In [26]:
from gluonts.evaluation import Evaluator
In [27]:
evaluator = Evaluator(quantiles=[0.1, 0.5, 0.9])
agg_metrics, item_metrics = evaluator(iter(tss), iter(forecasts), num_series=len(dataset.test))
Running evaluation: 100%|██████████| 414/414 [00:03<00:00, 130.54it/s]

Aggregate metrics aggregate both across time-steps and across time series.

In [28]:
print(json.dumps(agg_metrics, indent=4))
{
    "MSE": 5433799.990941212,
    "abs_error": 8127414.05900383,
    "abs_target_sum": 145558863.59960938,
    "abs_target_mean": 7324.822041043146,
    "seasonal_error": 336.9046924038305,
    "MASE": 2.63026660935021,
    "sMAPE": 0.181576173048337,
    "MSIS": 32.91678524157536,
    "QuantileLoss[0.1]": 4198629.754498768,
    "Coverage[0.1]": 0.1111111111111111,
    "QuantileLoss[0.5]": 8127413.99063921,
    "Coverage[0.5]": 0.4919987922705314,
    "QuantileLoss[0.9]": 5937776.170996856,
    "Coverage[0.9]": 0.8645330112721418,
    "RMSE": 2331.051263044468,
    "NRMSE": 0.3182399858976639,
    "ND": 0.055835926840978996,
    "wQuantileLoss[0.1]": 0.028844892373218797,
    "wQuantileLoss[0.5]": 0.055835926371309075,
    "wQuantileLoss[0.9]": 0.04079295498850535,
    "mean_wQuantileLoss": 0.04182459124434441,
    "MAE_Coverage": 0.018193102522812653
}

Individual metrics are aggregated only across time-steps.

In [29]:
item_metrics.head()
Out[29]:
item_id MSE abs_error abs_target_sum abs_target_mean seasonal_error MASE sMAPE MSIS QuantileLoss[0.1] Coverage[0.1] QuantileLoss[0.5] Coverage[0.5] QuantileLoss[0.9] Coverage[0.9]
0 NaN 1696.564128 1585.679688 31644.0 659.250000 42.371302 0.779655 0.048317 4.749242 775.275146 0.000000 1585.679718 0.750000 1156.547784 1.00
1 NaN 141520.437500 16027.132812 124149.0 2586.437500 165.107988 2.022304 0.123102 13.114531 4040.411304 0.333333 16027.133545 1.000000 7459.745654 1.00
2 NaN 27717.382812 6315.360352 65030.0 1354.791667 78.889053 1.667785 0.093389 12.333195 3038.133606 0.000000 6315.360474 0.104167 2072.121924 0.75
3 NaN 141436.937500 14291.365234 235783.0 4912.145833 258.982249 1.149642 0.059615 6.285117 8115.702686 0.020833 14291.364990 0.375000 5934.578076 1.00
4 NaN 90145.333333 11159.789062 131088.0 2731.000000 200.494083 1.159613 0.077624 6.205342 3867.461621 0.041667 11159.788574 0.708333 6444.027026 1.00
In [30]:
item_metrics.plot(x='MSIS', y='MASE', kind='scatter')
plt.grid(which="both")
plt.show()
../../_images/examples_basic_forecasting_tutorial_tutorial_46_0.png

Create your own forecast model

For creating your own forecast model you need to:

  • Define the training and prediction network
  • Define a new estimator that specifies any data processing and uses the networks

The training and prediction networks can be arbitrarily complex but they should follow some basic rules:

  • Both should have a hybrid_forward method that defines what should happen when the network is called
  • The trainng network’s hybrid_forward should return a loss based on the prediction and the true values
  • The prediction network’s hybrid_forward should return the predictions

For example, we can create a simple training network that defines a neural network which takes as an input the past values of the time series and outputs a future predicted window of length prediction_length. It uses the L1 loss in the hybrid_forward method to evaluate the error among the predictions and the true values of the time series. The corresponding prediction network should be identical to the training network in terms of architecture (we achieve this by inheriting the training network class), and its hybrid_forward method outputs directly the predictions.

Note that this simple model does only point forecasts by construction, i.e., we train it to outputs directly the future values of the time series and not any probabilistic view of the future (to achieve this we should train a network to learn a probability distribution and then sample from it to create sample paths).

In [31]:
class MyTrainNetwork(gluon.HybridBlock):
    def __init__(self, prediction_length, **kwargs):
        super().__init__(**kwargs)
        self.prediction_length = prediction_length

        with self.name_scope():
            # Set up a 3 layer neural network that directly predicts the target values
            self.nn = mx.gluon.nn.HybridSequential()
            self.nn.add(mx.gluon.nn.Dense(units=40, activation='relu'))
            self.nn.add(mx.gluon.nn.Dense(units=40, activation='relu'))
            self.nn.add(mx.gluon.nn.Dense(units=self.prediction_length, activation='softrelu'))

    def hybrid_forward(self, F, past_target, future_target):
        prediction = self.nn(past_target)
        # calculate L1 loss with the future_target to learn the median
        return (prediction - future_target).abs().mean(axis=-1)


class MyPredNetwork(MyTrainNetwork):
    # The prediction network only receives past_target and returns predictions
    def hybrid_forward(self, F, past_target):
        prediction = self.nn(past_target)
        return prediction.expand_dims(axis=1)

Now, we need to construct the estimator which should also follow some rules:

  • It should include a create_transformation method that defines all the possible feature transformations and how the data is split during training
  • It should include a create_training_network method that returns the training network configured with any necessary hyperparameters
  • It should include a create_predictor method that creates the prediction network, and returns a Predictor object

A Predictor defines the predict method of a given predictor. Roughly, this method takes the test dataset, it passes it through the prediction network and yields the predictions. You can think of the Predictor object as a wrapper of the prediction network that defines its predict method.

Earlier, we used the make_evaluation_predictions to evaluate our predictor. Internally, the make_evaluation_predictions function invokes the predict method of the predictor to get the forecasts.

In [32]:
from gluonts.model.estimator import GluonEstimator
from gluonts.model.predictor import Predictor, RepresentableBlockPredictor
from gluonts.core.component import validated
from gluonts.support.util import copy_parameters
from gluonts.transform import ExpectedNumInstanceSampler, Transformation, InstanceSplitter
from gluonts.dataset.field_names import FieldName
from mxnet.gluon import HybridBlock
In [33]:
class MyEstimator(GluonEstimator):
    @validated()
    def __init__(
        self,
        freq: str,
        context_length: int,
        prediction_length: int,
        trainer: Trainer = Trainer()
    ) -> None:
        super().__init__(trainer=trainer)
        self.context_length = context_length
        self.prediction_length = prediction_length
        self.freq = freq


    def create_transformation(self):
        # Feature transformation that the model uses for input.
        # Here we use a transformation that randomly select training samples from all time series.
        return InstanceSplitter(
                    target_field=FieldName.TARGET,
                    is_pad_field=FieldName.IS_PAD,
                    start_field=FieldName.START,
                    forecast_start_field=FieldName.FORECAST_START,
                    train_sampler=ExpectedNumInstanceSampler(num_instances=1),
                    past_length=self.context_length,
                    future_length=self.prediction_length,
                )

    def create_training_network(self) -> MyTrainNetwork:
        return MyTrainNetwork(
            prediction_length=self.prediction_length
        )

    def create_predictor(
        self, transformation: Transformation, trained_network: HybridBlock
    ) -> Predictor:
        prediction_network = MyPredNetwork(
            prediction_length=self.prediction_length
        )

        copy_parameters(trained_network, prediction_network)

        return RepresentableBlockPredictor(
            input_transform=transformation,
            prediction_net=prediction_network,
            batch_size=self.trainer.batch_size,
            freq=self.freq,
            prediction_length=self.prediction_length,
            ctx=self.trainer.ctx,
        )
INFO:root:Using CPU

Now, we can repeat the same pipeline as in the case we had a pre-built model: train the predictor, create the forecasts and evaluate the results.

In [34]:
estimator = MyEstimator(
    prediction_length=dataset.metadata.prediction_length,
    context_length=100,
    freq=dataset.metadata.freq,
    trainer=Trainer(ctx="cpu",
                    epochs=5,
                    learning_rate=1e-3,
                    num_batches_per_epoch=100
                   )
)
In [35]:
predictor = estimator.train(dataset.train)
INFO:root:Start model training
INFO:root:Number of parameters in MyTrainNetwork: 128
INFO:root:Epoch[0] Learning rate is 0.001
100%|██████████| 100/100 [00:00<00:00, 207.28it/s, avg_epoch_loss=1.99e+3]
INFO:root:Epoch[0] Elapsed time 0.484 seconds
INFO:root:Epoch[0] Evaluation metric 'epoch_loss'=1992.168558
INFO:root:Epoch[1] Learning rate is 0.001
100%|██████████| 100/100 [00:00<00:00, 192.40it/s, avg_epoch_loss=1.02e+3]
INFO:root:Epoch[1] Elapsed time 0.521 seconds
INFO:root:Epoch[1] Evaluation metric 'epoch_loss'=1023.022491
INFO:root:Epoch[2] Learning rate is 0.001
100%|██████████| 100/100 [00:00<00:00, 199.72it/s, avg_epoch_loss=760]
INFO:root:Epoch[2] Elapsed time 0.502 seconds
INFO:root:Epoch[2] Evaluation metric 'epoch_loss'=760.060089
INFO:root:Epoch[3] Learning rate is 0.001
100%|██████████| 100/100 [00:00<00:00, 125.04it/s, avg_epoch_loss=673]
INFO:root:Epoch[3] Elapsed time 0.802 seconds
INFO:root:Epoch[3] Evaluation metric 'epoch_loss'=672.758556
INFO:root:Epoch[4] Learning rate is 0.001
100%|██████████| 100/100 [00:00<00:00, 216.60it/s, avg_epoch_loss=636]
INFO:root:Epoch[4] Elapsed time 0.470 seconds
INFO:root:Epoch[4] Evaluation metric 'epoch_loss'=635.998240
INFO:root:Loading parameters from best epoch (4)
INFO:root:Final loss: 635.9982400512695 (occurred at epoch 4)
INFO:root:End model training
In [36]:
forecast_it, ts_it = make_evaluation_predictions(
    dataset=dataset.test,
    predictor=predictor,
    num_eval_samples=100
)
In [37]:
forecasts = list(forecast_it)
tss = list(ts_it)
In [38]:
plot_prob_forecasts(tss[0], forecasts[0])
../../_images/examples_basic_forecasting_tutorial_tutorial_57_0.png

Observe that we cannot actually see any prediction intervals in the predictions. This is expected since the model that we defined does not do probabilistic forecasting but it just gives point estimates. By requiring 100 sample paths (defined in make_evaluation_predictions) in such a network, we get 100 times the same output.

In [39]:
evaluator = Evaluator(quantiles=[0.1, 0.5, 0.9])
agg_metrics, item_metrics = evaluator(iter(tss), iter(forecasts), num_series=len(dataset.test))
Running evaluation: 100%|██████████| 414/414 [00:02<00:00, 198.64it/s]
In [40]:
print(json.dumps(agg_metrics, indent=4))
{
    "MSE": 13561622.756932631,
    "abs_error": 12322191.594608307,
    "abs_target_sum": 145558863.59960938,
    "abs_target_mean": 7324.822041043146,
    "seasonal_error": 336.9046924038305,
    "MASE": 4.987629351280599,
    "sMAPE": 0.22699351606791038,
    "MSIS": 199.50517568504125,
    "QuantileLoss[0.1]": 18392534.208760932,
    "Coverage[0.1]": 0.6371276167471819,
    "QuantileLoss[0.5]": 12322191.649746418,
    "Coverage[0.5]": 0.6371276167471819,
    "QuantileLoss[0.9]": 6251849.090731908,
    "Coverage[0.9]": 0.6371276167471819,
    "RMSE": 3682.61086145857,
    "NRMSE": 0.5027577244639954,
    "ND": 0.08465435418967764,
    "wQuantileLoss[0.1]": 0.1263580503029586,
    "wQuantileLoss[0.5]": 0.08465435456848047,
    "wQuantileLoss[0.9]": 0.04295065883400236,
    "mean_wQuantileLoss": 0.08465435456848047,
    "MAE_Coverage": 0.31237587224906066
}
In [41]:
item_metrics.head(10)
Out[41]:
item_id MSE abs_error abs_target_sum abs_target_mean seasonal_error MASE sMAPE MSIS QuantileLoss[0.1] Coverage[0.1] QuantileLoss[0.5] Coverage[0.5] QuantileLoss[0.9] Coverage[0.9]
0 NaN 4.316980e+03 2539.999023 31644.0 659.250000 42.371302 1.248879 0.078013 49.955180 4288.339661 0.812500 2539.999084 0.812500 791.658508 0.812500
1 NaN 2.767989e+05 23269.777344 124149.0 2586.437500 165.107988 2.936182 0.176607 117.447250 41870.762451 0.979167 23269.777100 0.979167 4668.791748 0.979167
2 NaN 5.470981e+04 9396.887695 65030.0 1354.791667 78.889053 2.481567 0.144835 99.262698 3373.795068 0.208333 9396.887939 0.208333 15419.980811 0.208333
3 NaN 2.182518e+05 16017.192383 235783.0 4912.145833 258.982249 1.288473 0.066158 51.538900 21168.597363 0.604167 16017.191895 0.604167 10865.786426 0.604167
4 NaN 1.115952e+05 13499.427734 131088.0 2731.000000 200.494083 1.402725 0.101239 56.109005 12597.147583 0.562500 13499.428345 0.562500 14401.709106 0.562500
5 NaN 2.096091e+05 18189.998047 303379.0 6320.395833 212.875740 1.780185 0.060751 71.207413 29218.563574 0.812500 18189.997559 0.812500 7161.431543 0.812500
6 NaN 8.830857e+06 117238.187500 1985325.0 41360.937500 1947.687870 1.254032 0.061092 50.161266 68444.241406 0.333333 117238.191406 0.333333 166032.141406 0.333333
7 NaN 5.240241e+06 91546.937500 1540706.0 32098.041667 1624.044379 1.174369 0.060388 46.974772 99811.509375 0.583333 91546.937500 0.583333 83282.365625 0.583333
8 NaN 1.740959e+07 169374.625000 1640860.0 34184.583333 1850.988166 1.906354 0.096137 76.254145 298830.980859 0.916667 169374.623047 0.916667 39918.265234 0.916667
9 NaN 7.016270e+02 995.095520 21408.0 446.000000 10.526627 1.969402 0.046737 78.776064 869.148889 0.541667 995.095520 0.541667 1121.042151 0.541667
In [42]:
item_metrics.plot(x='MSIS', y='MASE', kind='scatter')
plt.grid(which="both")
plt.show()
../../_images/examples_basic_forecasting_tutorial_tutorial_62_0.png