utils.eval

utils.eval

Functions

Name	Description
mo_cv_models	Performs cross-validation for separate models for each target in a multi-output problem.
mo_eval_models	Trains and evaluates separate models for each target in a multi-output regression problem.

mo_cv_models

utils.eval.mo_cv_models(X, y, model_define_func, cv=5, scores=None)

Performs cross-validation for separate models for each target in a multi-output problem.

Parameters

Name	Type	Description	Default
X	pd.DataFrame or np.ndarray	Feature matrix.	required
y	pd.DataFrame or np.ndarray	Target matrix with multiple columns.	required
model_define_func	Callable	Function that returns a fresh model or pipeline instance.	required
cv	int or cross-validation generator, default=5	Number of folds or CV object.	5
scores	Union[Dict[str, str / Callable], str, Callable, None]	Scoring metric(s). - If None (default): Uses default scorer of the estimator. Returns List[np.ndarray]. - If str/Callable: Single scorer. Returns List[np.ndarray]. - If Dict: Dictionary of {name: scorer}. Returns Dict[str, List[np.ndarray]]. Note: Unlike mo_eval_models which takes raw functions, cross_val_score expects strings (e.g. ‘neg_mean_squared_error’) or make_scorer callables, or raw callables that fit the sklearn signature. To maintain similarity with mo_eval_models, we rely on cross_val_score’s flexibility.	None

Returns

Name	Type	Description
scores	Union[List[np.ndarray], Dict[str, List[np.ndarray]]]	- List of arrays (one array of CV scores per target) if scores is None or single. - Dictionary of {name: List[np.ndarray]} if scores is a dictionary.

Examples

>>> import pandas as pd
>>> import numpy as np
>>> from spotoptim.utils.eval import mo_cv_models

>>> # Generate dummy data
>>> np.random.seed(42)
>>> X = pd.DataFrame(np.random.rand(100, 5), columns=[f'x{i}' for i in range(5)])
>>> y = pd.DataFrame(np.random.rand(100, 3), columns=[f'y{i}' for i in range(3)])

>>> # Example 1: Default behavior (Default scorer, e.g. R2)
>>> def make_model():
...     from sklearn.linear_model import Ridge
...     return Ridge()
>>> cv_scores = mo_cv_models(X, y, make_model, cv=3)
Cross-validating target 1/3...
Cross-validating target 2/3...
Cross-validating target 3/3...
CV Scores Mean: ['-0.14', '-0.07', '-0.20']

>>> # Example 2: Custom single score (NMSE - Negative Mean Squared Error)
>>> # Note: sklearn scoring strings are preferred for cross_val_score
>>> nmse_scores = mo_cv_models(X, y, make_model, cv=3, scores='neg_mean_squared_error')
Cross-validating target 1/3...
Cross-validating target 2/3...
Cross-validating target 3/3...
CV Scores Mean: ['-0.09', '-0.08', '-0.10']

>>> # Example 3: Multiple custom scores
>>> my_scores = {'R2': 'r2', 'NMSE': 'neg_mean_squared_error'}
>>> all_cv_scores = mo_cv_models(X, y, make_model, cv=3, scores=my_scores)
Cross-validating target 1/3...
Cross-validating target 2/3...
Cross-validating target 3/3...
CV Scores Mean:
  R2: ['-0.14', '-0.07', '-0.20']
  NMSE: ['-0.09', '-0.08', '-0.10']

mo_eval_models

utils.eval.mo_eval_models(
    X_train,
    y_train,
    X_test,
    y_test,
    model_define_func,
    scores=None,
    verbose=False,
)

Trains and evaluates separate models for each target in a multi-output regression problem.

Parameters

Name	Type	Description	Default
X_train	pd.DataFrame or np.ndarray	Training feature matrix.	required
y_train	pd.DataFrame or np.ndarray	Training target matrix with multiple columns (one per target).	required
X_test	pd.DataFrame or np.ndarray	Test feature matrix.	required
y_test	pd.DataFrame or np.ndarray	Test target matrix with multiple columns (one per target).	required
model_define_func	Callable	Function that returns a fresh model or pipeline instance for training.	required
scores	Union[Dict[str, Callable], Callable, None]	Scoring metric(s) to evaluate. - If None (default): Uses R2 score. Returns List[float]. - If Callable: Single scoring function (e.g., mean_squared_error). Returns List[float]. - If Dict: Dictionary of {name: scoring_func}. Returns Dict[str, List[float]].	None
verbose	bool	Whether to print verbose output.	False

Returns

Name	Type	Description
scores	Union[List[float], Dict[str, List[float]]]	- List of scores if scores is None or a single callable. - Dictionary of scores if scores is a dictionary.
models	List	List of trained model instances, one per target.
preds_stacked	np.ndarray	Array of stacked predictions for all targets, shape (n_samples, n_targets).

Examples

>>> import pandas as pd
>>> import numpy as np
>>> from spotoptim.utils.eval import mo_eval_models

>>> # Generate dummy data
>>> np.random.seed(42)
>>> X_train = pd.DataFrame(np.random.rand(100, 5), columns=[f'x{i}' for i in range(5)])
>>> y_train = pd.DataFrame(np.random.rand(100, 3), columns=[f'y{i}' for i in range(3)])
>>> X_test = pd.DataFrame(np.random.rand(20, 5), columns=[f'x{i}' for i in range(5)])
>>> y_test = pd.DataFrame(np.random.rand(20, 3), columns=[f'y{i}' for i in range(3)])

>>> # Example 1: Default behavior (R2 score)
>>> def make_model():
...     from sklearn.linear_model import Ridge
...     return Ridge()
>>> r2_scores, models, preds = mo_eval_models(X_train, y_train, X_test, y_test, make_model)
Training model for target 1/3...
Training model for target 2/3...
Training model for target 3/3...
Model scores: ['-0.10', '-0.13', '-0.19']
Predictions shape: (20, 3)

>>> # Example 2: Custom single score (MSE)
>>> from sklearn.metrics import mean_squared_error
>>> mse_scores, _, _ = mo_eval_models(X_train, y_train, X_test, y_test, make_model, scores=mean_squared_error)
Training model for target 1/3...
Training model for target 2/3...
Training model for target 3/3...
Model scores: ['0.07', '0.09', '0.10']
Predictions shape: (20, 3)

>>> # Example 3: Multiple custom scores
>>> from sklearn.metrics import mean_absolute_error, r2_score
>>> my_scores = {'R2': r2_score, 'MSE': mean_squared_error, 'MAE': mean_absolute_error}
>>> all_scores, _, _ = mo_eval_models(X_train, y_train, X_test, y_test, make_model, scores=my_scores)
Training model for target 1/3...
Training model for target 2/3...
Training model for target 3/3...
Model scores:
  R2: ['-0.10', '-0.13', '-0.19']
  MSE: ['0.07', '0.09', '0.10']
  MAE: ['0.21', '0.27', '0.28']
Predictions shape: (20, 3)