Table Of Contents
Table Of Contents

gluonts.evaluation package


Returns the default seasonality for a given freq str. E.g. for

2H -> 12

class gluonts.evaluation.Evaluator(quantiles: Iterable[Union[float, str]] = (0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9), seasonality: Optional[int] = None, alpha: float = 0.05)[source]

Bases: object

Evaluator class, to compute accuracy metrics by comparing observations to forecasts.

  • quantiles – list of strings of the form ‘p10’ or floats in [0, 1] with the quantile levels
  • seasonality – seasonality to use for seasonal_error, if nothing is passed uses the default seasonality for the given series frequency as returned by get_seasonality
  • alpha – parameter of the MSIS metric from M4 competition that defines the confidence interval for alpha=0.05 the 95% considered is considered in the metric, see -Competitors-Guide.pdf for more detail on MSIS
static abs_error(target, forecast)[source]
static abs_target_mean(target)[source]
static abs_target_sum(target)[source]
static coverage(target, quantile_forecast)[source]
default_quantiles = (0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9)
static extract_pred_target(time_series: Union[pandas.core.series.Series, pandas.core.frame.DataFrame], forecast: gluonts.model.forecast.Forecast) → Union[pandas.core.series.Series, pandas.core.frame.DataFrame][source]
  • time_series
  • forecast

time series cut in the Forecast object dates

Return type:

Union[pandas.Series, pandas.DataFrame]

get_aggregate_metrics(metric_per_ts: pandas.core.frame.DataFrame) → Tuple[Dict[str, float], pandas.core.frame.DataFrame][source]
get_metrics_per_ts(time_series: Union[pandas.core.series.Series, pandas.core.frame.DataFrame], forecast: gluonts.model.forecast.Forecast) → Dict[str, float][source]
static mase(target, forecast, seasonal_error)[source]
\[mase = mean(|Y - Y_hat|) / seasonal_error\]

static mse(target, forecast)[source]
static msis(target, lower_quantile, upper_quantile, seasonal_error, alpha)[source]
Math:msis = mean(U - L + 2/alpha * (L-Y) * I[Y<L] + 2/alpha * (Y-U) * I[Y>U]) /seasonal_error

static quantile_loss(target, quantile_forecast, q)[source]
seasonal_error(time_series: Union[pandas.core.series.Series, pandas.core.frame.DataFrame], forecast: gluonts.model.forecast.Forecast) → float[source]
\[seasonal_error = mean(|Y[t] - Y[t-m]|)\]

where m is the seasonal frequency

static smape(target, forcecast)[source]
\[smape = mean(2 * |Y - Y_hat| / (|Y| + |Y_hat|))\]

class gluonts.evaluation.MultivariateEvaluator(quantiles: Iterable[Union[float, str]], seasonality: Optional[int] = None, alpha: float = 0.05, eval_dims: List[int] = None)[source]

Bases: gluonts.evaluation._base.Evaluator

The MultivariateEvaluator class owns functionality for evaluating multidimensional target arrays of shape (target_dimensionality, prediction_length).

The aggregate metric keys in the output dictionary correspond to the aggregated metrics over the entire target array. Additionally, evaluations of individual dimensions will be stored in the corresponding dimension key and contain the metrics calculated by only this dimension. Evaluation dimensions can be set by the user.


{0: {‘MSE’: 0.004307240342677687, ‘abs_error’: 1.6246897801756859, ‘abs_target_sum’: 90.0, …}, 1: {‘MSE’: 0.003949341769475723, ‘abs_error’: 1.5052175521850586, ‘abs_target_sum’: 290.0,…}, MSE’: 0.004128291056076705, ‘abs_error’: 3.1299073323607445, ‘abs_target_sum’: 380.0, …}

calculate_aggregate_vector_metrics(all_agg_metrics: Dict[int, Dict[str, float]], all_metrics_per_ts: pandas.core.frame.DataFrame)[source]
  • all_agg_metrics – dictionary with aggregate metrics of individual dimensions
  • all_metrics_per_ts – DataFrame containing metrics for all time series of all evaluated dimensions

dictionary with aggregate metrics (of individual (evaluated) dimensions and the entire vector)

Return type:

Dict[int, Dict[str, float]]

static extract_forecast_by_dim(forecast_iterator: Iterator[gluonts.model.forecast.Forecast], dim: int) → Iterator[gluonts.model.forecast.Forecast][source]
static extract_target_by_dim(it_iterator: Iterator[pandas.core.frame.DataFrame], dim: int) → Iterator[pandas.core.frame.DataFrame][source]
get_eval_dims(target_dimensionality: int) → List[int][source]
static get_target_dimensionality(forecast: gluonts.model.forecast.Forecast) → int[source]
static peek(iterator: Iterator[Any]) → Tuple[Any, Iterator[Any]][source]