n2n_predict_with_covariates on demo10: Step by Step on Real Data
Sibling walkthrough that executes each pipeline stage from n2n_predict_with_covariates.py against the bundled demo10.csv, one helper at a time.
What this page does
This is the executable companion to n2n_predict_with_covariates: A Beginner’s Walkthrough. The sibling page introduces the vocabulary and explains the why of each stage with small synthetic examples. This page does the opposite: every code cell executes one real stage of spotforecast2_safe.processing.n2n_predict_with_covariates against the bundled demo10.csv dataset, prints the intermediate state, and then discusses what just happened. Refer back to the sibling page for the vocabulary — @def-time-series, @def-lag, @def-forecast-horizon, @def-recursive-forecaster, @def-train-val-test, @def-outlier, @def-imputation, @def-sample-weight, @def-cyclical-encoding, @def-persistence — all definitions are reused unchanged.
The closing section calls n2n_predict_with_covariates end-to-end on the same input as a sanity check that the per-stage breakdown above is equivalent to the orchestrator.
About demo10.csv
demo10.csv is bundled in the wheel at spotforecast2_safe/datasets/csv/demo10.csv. It carries eleven numeric columns A–K, indexed by hourly UTC timestamps from December 2019 through December 2021 — about 18 000 rows, roughly two years. Column C comes online late (mid-May 2021) and starts with a long run of NaN covering most of the window, which makes it a useful exercise for the imputation step in Stage 3.
demo10.csv is the compact sibling of the much larger demo100.csv (about 96 000 rows spanning 2010–2021). Its two-year span already covers every helper faithfully — the 7-day rolling weather window in Stage 4 and the 80/20 temporal split in Stage 8 both have ample data — while keeping render times short, so no row slicing is needed before Stage 1 begins.
Setup
Example 1 (Loading demo10)
from pathlib import Pathimport pandas as pdfrom spotforecast2_safe.data.fetch_data import fetch_data, get_package_data_homedata_demo = fetch_data( filename=get_package_data_home() /"demo10.csv", timezone="UTC",)cache_home = Path.home() /".spotforecast2_cache"print("shape :", data_demo.shape)print("window :", data_demo.index[0], "→", data_demo.index[-1])print("NaN per col :")print(data_demo.isna().sum())
shape : (18118, 11)
window : 2019-12-01 00:00:00+00:00 → 2021-12-24 21:00:00+00:00
NaN per col :
A 0
B 0
C 12707
D 0
E 0
F 0
G 0
H 0
I 0
J 0
K 0
dtype: int64
demo10 is loaded whole — its two-year span is already small enough to render quickly, so no row slicing is required and every stage below sees the full series. cache_home points at the same directory that get_cache_home() would resolve to (~/.spotforecast2_cache); reusing it means subsequent renders read the weather parquet cache instead of refetching from Open-Meteo.
Stage 1 — Loading and preparing the target series
A forecasting model needs a clean, regularly spaced time series with a known time zone. Stage 1 turns whatever the caller supplied into exactly that. get_start_end returns four boundary timestamps: start and end delimit the historical window used for training, and cov_start / cov_end are the same window extended forward by forecast_horizon steps — that extension is the future window for which covariates must be constructed in Stage 4.
cov_end lies exactly forecast_horizon hours past end — that is the window for which Stage 4 will assemble covariates and Stage 10 will emit predictions. basic_ts_checks confirms the index is a strict, gap-free, timezone-aware DatetimeIndex; if it were not, agg_and_resample_data could not safely enforce the hourly grid.
Why a strict, gap-free hourly index matters
A recursive forecaster (see @def-recursive-forecaster in the sibling page) looks up lag 1, lag 24, etc. by positional offset. A skipped hour would silently turn “lag 24” into “25 wall-clock hours ago”, which is the kind of error that compounds for hundreds of forecast steps before it is noticed.
Stage 2 — Outlier detection and removal
Stage 2 replaces likely sensor glitches with NaN so Stage 3 can treat them the same way it treats genuine gaps. An Isolation Forest is applied per column with contamination=0.01, meaning about one percent of rows per column should be flagged. The random_state=1234 is fixed so two runs on identical input produce identical outlier flags.
Example 3 (Marking outliers with Isolation Forest)
from spotforecast2_safe.preprocessing.outlier import mark_outliersdata, outliers = mark_outliers( data, contamination=0.01, random_state=1234, verbose=False,)print("outliers flagged :", int((outliers ==-1).sum()) ifhasattr(outliers, "__iter__") else"—")print("NaN per column after outlier removal:")print(data.isna().sum())
outliers flagged : 181
NaN per column after outlier removal:
A 182
B 181
C 12889
D 181
E 177
F 172
G 182
H 177
I 182
J 181
K 181
dtype: int64
The per-column NaN counts now combine pre-existing gaps (notably the long leading run in column C) with the freshly removed outlier rows. Stage 3 will not distinguish between the two: imputation closes the gaps, and sample weighting penalises the model for paying attention to either source.
Stage 3 — Imputation and sample weighting
get_missing_weights forward- and backward-fills every NaN, then builds a per-row weight series: rows whose window_size=72 neighbourhood touched an imputed cell receive weight 0; all other rows receive 1. The weights are wrapped in a WeightFunction, which is a picklable class that the forecaster can carry into a joblib dump alongside the model itself — a closure would survive pickle.dumps but not a pickle.load in a fresh process.
Example 4 (Imputation count and the weight distribution)
from spotforecast2_safe.preprocessing import WeightFunctionfrom spotforecast2_safe.preprocessing.imputation import get_missing_weightsimputed_data, weights_series = get_missing_weights( data, window_size=72, verbose=False,)weight_func = WeightFunction(weights_series)print("imputed NaN remaining :", int(imputed_data.isna().sum().sum()))print("weight distribution :")print(weights_series.value_counts(dropna=False).sort_index())print("weight_func on first 5 timestamps :", weight_func(imputed_data.index[:5]))
imputed NaN remaining : 0
weight distribution :
0.0 18055
1.0 63
Name: count, dtype: int64
weight_func on first 5 timestamps : None
The imputed NaN remaining count is zero — every gap is closed. The weight distribution shows how many rows the forecaster will effectively ignore; the bulk of the zero-weighted rows are concentrated at the start of the window where column C had its leading NaN run.
Stage 4 — Exogenous feature engineering
Stage 4 builds four feature DataFrames, each indexed on the extended timeline [start, cov_end] so the feature matrix is defined both over the training window and over the future prediction window. The sub-stages are independent and can run in any order.
Stage 4a — Calendar features
Example 5 (Calendar features from the index alone)
Calendar features are derived from the index alone, so they are always complete by construction — no imputation is needed and the missing-value check at the end of Stage 5 cannot fail on these columns.
Stage 4b — Day/night features
Example 6 (Sunrise, sunset, and the daylight flag)
is_daylight.mean() averaged over two years at 51° N should land close to 0.5. Per-day sunrise and sunset values are cached internally so the solar position is not recomputed for every hourly row.
Stage 4c — Weather features
First render needs network
Stage 4c reaches Open-Meteo. The helper defaults to fallback_on_failure=True, so renders without network will still succeed (with degraded weather data); but the page is most informative when run locally once with cache_home pointed at a writable directory.
Example 7 (Open-Meteo fetch and rolling weather windows)
weather_aligned carries the raw weather columns aligned to the extended timeline. weather_features carries the same plus rolling one-day and seven-day mean/min/max windows for each numeric weather column. Whether the windows survive into the final feature matrix is decided by the include_weather_windows flag in Stage 6.
Stage 4d — Holiday features
Example 8 (German public holidays for North Rhine-Westphalia)
For two years of NRW the expected count is roughly eleven holidays per year × twenty-four hours ≈ five hundred flagged hours. The model sees a clean binary column with no NaN.
Stage 5 — Combining and encoding exogenous features
The four feature DataFrames are concatenated column-wise in a fixed order: calendar, day/night, weather, holidays. Any NaN that survived into the concatenated matrix would be a bug — Stage 5 raises before any further work happens. Then apply_cyclical_encoding replaces every periodic integer column (month, week, day_of_week, hour, sunrise_hour, sunset_hour) with its sine and cosine on a unit circle, keeping the original integers alongside, and create_interaction_features emits pairwise products of cyclical, weather, and holiday columns prefixed with poly_.
Example 9 (Concat, NaN guard, cyclical encoding, interactions)
import pandas as pdfrom spotforecast2_safe.manager.features import ( apply_cyclical_encoding, create_interaction_features,)exogenous_features = pd.concat( [calendar_features, sun_light_features, weather_features, holiday_features], axis=1,)missing_count =int(exogenous_features.isnull().sum().sum())print("missing entries in concat :", missing_count)print("shape after concat :", exogenous_features.shape)exogenous_features = apply_cyclical_encoding( data=exogenous_features, drop_original=False,)sin_cos_cols = [c for c in exogenous_features.columns if c.endswith("_sin") or c.endswith("_cos")]print("shape after cyclical enc :", exogenous_features.shape)print("number of sin/cos columns :", len(sin_cos_cols))exogenous_features = create_interaction_features( exogenous_features=exogenous_features, weather_aligned=weather_aligned,)poly_cols = [c for c in exogenous_features.columns if c.startswith("poly_")]print("shape after interactions :", exogenous_features.shape)print("number of poly_ columns :", len(poly_cols))
missing entries in concat : 0
shape after concat : (18142, 114)
shape after cyclical enc : (18142, 126)
number of sin/cos columns : 12
shape after interactions : (18142, 126)
number of poly_ columns : 0
Why sine and cosine
Feeding the integer hour 0–23 directly to a tree-based model makes hour 23 and hour 0 look very far apart numerically, even though they are neighbours on a clock. Sine and cosine wrap the integer onto a unit circle, so adjacent hours stay adjacent in the encoded space. See @def-cyclical-encoding in the sibling page.
Stage 6 — Feature selection
select_exogenous_features returns the final list of column names that will be passed to the forecaster as exog. With the three include flags set to False (the package defaults), the selection keeps only the cyclical sine/cosine columns and the raw weather columns; rolling weather windows, the holiday column, and the polynomial interactions are filtered out.
Example 10 (Final exogenous column list)
from spotforecast2_safe.manager.features import select_exogenous_featuresexog_features = select_exogenous_features( exogenous_features=exogenous_features, weather_aligned=weather_aligned, include_weather_windows=False, include_holiday_features=False, poly_features_degree=1,)print("number of exog features :", len(exog_features))print("first 6 :", exog_features[:6])print("last 3 :", exog_features[-3:])
number of exog features : 27
first 6 : ['month_sin', 'month_cos', 'week_sin', 'week_cos', 'day_of_week_sin', 'day_of_week_cos']
last 3 : ['wind_speed_10m', 'wind_direction_10m', 'wind_gusts_10m']
To experiment with the other flags, re-run this cell with one or more of the include_* arguments set to True and observe how len(exog_features) grows.
Stage 7 — Merging target and exogenous data
merge_data_and_covariates performs an inner join of the target columns and the selected exogenous columns over [start, end], casts every column to float32 to halve the memory footprint of the lag matrix, and also returns the slice of the exogenous matrix that covers the future prediction window (end, cov_end].
Example 11 (Merged training matrix and the future-window slice)
exo_pred has exactly forecast_horizon rows; it is the matrix passed to every forecaster at prediction time. Every column of data_with_exog is float32, including the eleven target columns.
Stage 8 — Train / validation / test split
The split is purely temporal: the first train_ratio=0.8 fraction of rows is training data, and the remaining twenty percent goes to validation. The test segment is empty because perc_val = 1 - train_ratio consumes all remaining rows. end_validation is the cutoff timestamp used by forecaster.fit(...).
Example 12 (Temporal split and the validation cutoff)
train rows : 14494
val rows : 3624
test rows : 0
end_validation: 2021-12-24 21:00:00+00:00
Why temporal, not random
Random shuffling before a time-series split lets the model “see the future”: its training set could include hour 14 of a day on which the test set asks it to predict hour 13. A temporal split forces a real future-from-past forecast. See @def-train-val-test in the sibling page.
Stage 9 — Training recursive forecasters
For each of the eleven target columns Stage 9 builds a fresh ForecasterRecursive, configured identically: lags=24, RollingFeatures(stats=["mean"], window_sizes=72), the shared weight_func, and an LGBMRegressor with random_state=1234. Every fit uses data up to and including end_validation, so the test segment is held out for any downstream evaluation step.
This is the most expensive cell on the page — eleven LightGBM models on roughly fourteen thousand training rows each. Expect a few minutes on a modern laptop.
Example 13 (Eleven forecasters, one shared configuration)
from lightgbm import LGBMRegressorfrom spotforecast2_safe.forecaster.recursive import ForecasterRecursivefrom spotforecast2_safe.preprocessing import RollingFeaturesestimator = LGBMRegressor(random_state=1234, verbose=-1)window_features = RollingFeatures(stats=["mean"], window_sizes=72)recursive_forecasters = {}for target in target_columns: forecaster = ForecasterRecursive( estimator=estimator, lags=24, window_features=window_features, weight_func=weight_func, ) forecaster.fit( y=data_with_exog[target].loc[:end_validation].squeeze(), exog=data_with_exog[exog_features].loc[:end_validation], ) recursive_forecasters[target] = forecasterfeature_counts = {t: recursive_forecasters[t].estimator.n_features_in_ for t in target_columns}print("forecasters trained :", len(recursive_forecasters))print("estimator type :", type(recursive_forecasters[target_columns[0]].estimator).__name__)print("n_features_in_ across all :", set(feature_counts.values()))
forecasters trained : 11
estimator type : LGBMRegressor
n_features_in_ across all : {52}
All eleven forecasters share the same hyperparameters and random_state=1234, so two renders on identical input produce byte- identical models — the determinism guarantee underpins reproducible reporting.
Stage 10 — Prediction
predict_multivariate iterates over the trained forecasters and calls .predict(steps=forecast_horizon, exog=exo_pred[exog_features]) on each. The result is a single DataFrame with one column per target and forecast_horizon rows.
Example 14 (Twenty-four-hour multi-target forecast)
predictions shape : (24, 11)
first timestamp : 2021-12-24 22:00:00+00:00
last timestamp : 2021-12-25 21:00:00+00:00
A
B
C
D
E
F
G
H
I
J
K
2021-12-24 22:00:00+00:00
-783.34
4188.96
-497.04
72.88
1460.07
-35.50
44.40
8543.76
-108.32
-400.02
-226.03
2021-12-24 23:00:00+00:00
-1911.23
3427.85
-91.57
-381.16
1314.43
129.45
39.64
7877.67
-178.98
-398.70
-498.15
2021-12-25 00:00:00+00:00
-1907.70
4048.50
-403.56
-378.69
1345.64
122.53
47.88
7864.84
-68.93
-398.86
-499.13
predictions.shape is (24, 11): twenty-four forecast hours, eleven target columns. The index is exactly exo_pred.index, which is exactly forecast_horizon hours starting one step after end_validation.
Aggregation
Stage 10 produced one forecast column per target. The production task closes the pipeline with a weighted aggregation: agg_predict(predictions, weights=weights) reduces the eleven columns to a single Series sharing the DatetimeIndex of exo_pred. The conceptual sibling page explains the helper’s list / np.ndarray / dict accepted forms and the signed-weights convention in its Aggregation section.
Example 15 (Aggregating the eleven per-target forecasts)
The sign pattern of weights is identical to the module-level DEFAULT_WEIGHTS constant in tasks/task_safe_n_to_1_with_covariates_and_dataframe.py. With this signed choice the aggregate behaves as a net position — the first, second, fifth, seventh, eighth, ninth, and eleventh columns are added and the remainder are subtracted — and the call mirrors what the production task docs/tasks/task_safe_n21_cov_df.qmd executes after its own Stage 10.
Metadata
The orchestrator emits a metadata dictionary that records every configuration knob and every shape it just produced — a self-contained audit record. Building the same dictionary by hand is a useful integration check that nothing was lost along the way.
Example 16 (Reconstructing the orchestrator’s metadata)
The pipeline above is exactly what n2n_predict_with_covariates does internally. Calling the orchestrator on the same data with the same parameters should yield the same shapes and (modulo any change in Open-Meteo’s historical data between the two calls) the same numerical predictions.
Numerical equality of the two prediction matrices depends on Open-Meteo returning identical historical values between the two render passes; sharing the same cache_home is what makes that the common case rather than a rare one.
What to take away
Each stage of n2n_predict_with_covariates produces a checkable intermediate artefact: the boundary timestamps, the outlier mask, the weight series, the four exogenous DataFrames, the merged matrix, the three split DataFrames, the dictionary of fitted forecasters, and finally the predictions DataFrame.
The orchestrator is exactly the composition of these calls. There is no hidden state — the cross-check cell above proves it.
The only network-dependent stage is 4c. freeze: true keeps local iteration cheap, and cache_home keeps the network round-trip amortised across renders.
The same configuration on the same input yields byte-identical forecasters, predictions, and metadata. That determinism is the foundation that makes n2n_predict_with_covariates safe to embed in an automated batch job.