backtesting.validation.backtesting_forecaster_one_step

backtesting.validation.backtesting_forecaster_one_step(
    forecaster,
    y,
    cv,
    metric,
    exog=None,
    interval=None,
    interval_method='bootstrapping',
    n_boot=250,
    use_in_sample_residuals=True,
    use_binned_residuals=True,
    random_state=123,
    return_predictors=False,
    n_jobs='auto',
    verbose=False,
    show_progress=True,
    suppress_warnings=False,
)

Backtesting of forecaster model using one-step-ahead predictions.

Parameters

Name	Type	Description	Default
forecaster	(`ForecasterRecursive`, `ForecasterDirect`, `ForecasterEquivalentDate`)	Forecaster model.	required
y	pd.Series	Training time series.	required
cv	OneStepAheadFold	OneStepAheadFold object with the information needed to split the data into folds.	required
metric	str \| Callable \| list	Metric used to quantify the goodness of fit of the model.	required
exog	pd.Series \| pd.DataFrame	Exogenous variable/s included as predictor/s. Defaults to None.	`None`
interval	float \| list \| tuple \| str \| object	Specifies whether probabilistic predictions should be estimated.	`None`
interval_method	str	Technique used to estimate prediction intervals.	`'bootstrapping'`
n_boot	int	Number of bootstrapping iterations.	`250`
use_in_sample_residuals	bool	Use residuals from training data.	`True`
use_binned_residuals	bool	Use binned residuals for intervals.	`True`
random_state	int	Seed for reproducibility.	`123`
return_predictors	bool	Return predictors used for each prediction.	`False`
n_jobs	int \| str	Number of jobs to run in parallel.	`'auto'`
verbose	bool	Print information about the process.	`False`
show_progress	bool	Whether to show a progress bar.	`True`
suppress_warnings	bool	Suppress spotforecast warnings.	`False`

Returns

Name	Type	Description
tuple	(pd.DataFrame, pd.DataFrame)	- metric_values: Value(s) of the metric(s). - backtest_predictions: Value of predictions.

Notes

This function is designed for one-step-ahead backtesting, where predictions are made for the next time step using the most recent data. The function handles the fitting and prediction process for each fold defined in the OneStepAheadFold object, and calculates the specified metric(s) based on the true and predicted values. Depending on the interval and interval_method parameters, it can also generate probabilistic predictions.

Examples

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

from spotforecast2_safe.backtesting.validation import backtesting_forecaster_one_step
from spotforecast2_safe.forecaster import ForecasterRecursive
from spotforecast2_safe.splitter.split_one_step import OneStepAheadFold

rng = np.random.default_rng(0)
y = pd.Series(rng.standard_normal(100), name="y")
forecaster = ForecasterRecursive(estimator=LinearRegression(), lags=3)
cv = OneStepAheadFold(initial_train_size=50)

metric_values, backtest_predictions = backtesting_forecaster_one_step(
    forecaster=forecaster,
    y=y,
    cv=cv,
    metric="mean_squared_error",
    n_jobs=1,
    verbose=False,
    show_progress=False,
    suppress_warnings=True,
)
print(metric_values)
assert "mean_squared_error" in metric_values.columns
assert backtest_predictions.shape[0] == 50
assert "fold" in backtest_predictions.columns
assert "pred" in backtest_predictions.columns

   mean_squared_error
0            1.222658