Table Of Contents
Table Of Contents

gluonts.evaluation package


Returns the default seasonality for a given freq str. E.g. for

2H -> 12

class gluonts.evaluation.Evaluator(quantiles: Iterable[Union[float, str]] = (0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9), seasonality: Optional[int] = None, alpha: float = 0.05)[source]

Bases: object

Evaluator class, to compute accuracy metrics by comparing observations to forecasts.

  • quantiles – list of strings of the form ‘p10’ or floats in [0, 1] with the quantile levels
  • seasonality – seasonality to use for seasonal_error, if nothing is passed uses the default seasonality for the given series frequency as returned by get_seasonality
  • alpha – parameter of the MSIS metric from M4 competition that defines the confidence interval for alpha=0.05 the 95% considered is considered in the metric, see -Competitors-Guide.pdf for more detail on MSIS
static abs_error(target, forecast)[source]
static abs_target_mean(target)[source]
static abs_target_sum(target)[source]
static coverage(target, quantile_forecast)[source]
default_quantiles = (0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9)
static extract_pred_target(time_series: Union[pandas.core.series.Series, pandas.core.frame.DataFrame], forecast: gluonts.model.forecast.Forecast) → Union[pandas.core.series.Series, pandas.core.frame.DataFrame][source]
  • time_series
  • forecast

time series cut in the Forecast object dates

Return type:

Union[pandas.Series, pandas.DataFrame]

get_aggregate_metrics(metric_per_ts: pandas.core.frame.DataFrame) → Tuple[Dict[str, float], pandas.core.frame.DataFrame][source]
get_metrics_per_ts(time_series: Union[pandas.core.series.Series, pandas.core.frame.DataFrame], forecast: gluonts.model.forecast.Forecast) → Dict[str, Union[float, str, None]][source]
static mase(target, forecast, seasonal_error)[source]
\[mase = mean(|Y - Y_hat|) / seasonal_error\]

static mse(target, forecast)[source]
static msis(target, lower_quantile, upper_quantile, seasonal_error, alpha)[source]
Math:msis = mean(U - L + 2/alpha * (L-Y) * I[Y<L] + 2/alpha * (Y-U) * I[Y>U]) /seasonal_error

static quantile_loss(target, quantile_forecast, q)[source]
seasonal_error(time_series: Union[pandas.core.series.Series, pandas.core.frame.DataFrame], forecast: gluonts.model.forecast.Forecast) → float[source]
\[seasonal_error = mean(|Y[t] - Y[t-m]|)\]

where m is the seasonal frequency

static smape(target, forcecast)[source]
\[smape = mean(2 * |Y - Y_hat| / (|Y| + |Y_hat|))\]

class gluonts.evaluation.MultivariateEvaluator(quantiles: Iterable[Union[float, str]] = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]), seasonality: Optional[int] = None, alpha: float = 0.05, eval_dims: List[int] = None, target_agg_funcs: Dict[str, Callable] = {})[source]

Bases: gluonts.evaluation._base.Evaluator

The MultivariateEvaluator class owns functionality for evaluating multidimensional target arrays of shape (target_dimensionality, prediction_length).

Evaluations of individual dimensions will be stored with the corresponding dimension prefix and contain the metrics calculated by only this dimension. Metrics with the plain metric name correspond to metrics calculated over all dimensions. Additionally, the user can provide additional aggregation functions that first aggregate the target and forecast over dimensions and then calculate the metric. These metrics will be prefixed with m_<aggregation_fun_name>_

The evaluation dimensions can be set by the user.


{‘0_MSE’: 0.004307240342677687, # MSE of dimension 0 ‘0_abs_error’: 1.6246897801756859, ‘1_MSE’: 0.003949341769475723, # MSE of dimension 1 ‘1_abs_error’: 1.5052175521850586, ‘MSE’: 0.004128291056076705, # MSE of all dimensions ‘abs_error’: 3.1299073323607445, ‘m_sum_MSE’: 0.02 # MSE of aggregated target and aggregated forecast (if target_agg_funcs is set). ‘m_sum_abs_error’: 4.2}

calculate_aggregate_multivariate_metrics(ts_iterator: Iterator[pandas.core.frame.DataFrame], forecast_iterator: Iterator[gluonts.model.forecast.Forecast], agg_fun: Callable) → Dict[str, float][source]
  • ts_iterator – Iterator over time series
  • forecast_iterator – Iterator over forecasts
  • agg_fun – aggregation function

dictionary with aggregate datasets metrics

Return type:

Dict[str, float]

calculate_aggregate_vector_metrics(all_agg_metrics: Dict[str, float], all_metrics_per_ts: pandas.core.frame.DataFrame) → Dict[str, float][source]
  • all_agg_metrics – dictionary with aggregate metrics of individual dimensions
  • all_metrics_per_ts – DataFrame containing metrics for all time series of all evaluated dimensions

dictionary with aggregate metrics (of individual (evaluated) dimensions and the entire vector)

Return type:

Dict[str, float]

static extract_aggregate_forecast(forecast_iterator: Iterator[gluonts.model.forecast.Forecast], agg_fun: Callable) → Iterator[gluonts.model.forecast.Forecast][source]
static extract_aggregate_target(it_iterator: Iterator[pandas.core.frame.DataFrame], agg_fun: Callable) → Iterator[pandas.core.frame.DataFrame][source]
static extract_forecast_by_dim(forecast_iterator: Iterator[gluonts.model.forecast.Forecast], dim: int) → Iterator[gluonts.model.forecast.Forecast][source]
static extract_target_by_dim(it_iterator: Iterator[pandas.core.frame.DataFrame], dim: int) → Iterator[pandas.core.frame.DataFrame][source]
get_eval_dims(target_dimensionality: int) → List[int][source]
static get_target_dimensionality(forecast: gluonts.model.forecast.Forecast) → int[source]
static peek(iterator: Iterator[Any]) → Tuple[Any, Iterator[Any]][source]