First, we import all necessary packages for the examples.
# Core imports
import numpy as np
import time
from scipy.optimize import OptimizeResult
# SpotOptim
from spotoptim import SpotOptim
# Surrogate models
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF, ConstantKernel
# Set random seed for reproducibility
np.random.seed(42)
print("All packages imported successfully!")All packages imported successfully!optimize()The optimize() method is the main entry point for running the optimization process. It coordinates all other methods in the optimization workflow:
get_initial_design(), _curate_initial_design(), _rm_NA_values(), _check_size_initial_design(), _get_best_xy_initial_design()_determine_termination()Let’s see a complete optimization example:
# Define a simple quadratic function
def sphere(X):
"""Sphere function: f(x) = sum(x^2)"""
X = np.atleast_2d(X)
return np.sum(X**2, axis=1)
# Create optimizer
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
n_initial=5,
max_iter=20,
verbose=True
)
# Run optimization
result = opt.optimize()
print(f"\nBest point found: {result.x}")
print(f"Best value: {result.fun:.6f}")
print(f"Total evaluations: {result.nfev}")
print(f"Sequential iterations: {result.nit}")
print(f"Success: {result.success}")
print(f"Message: {result.message}")TensorBoard logging disabled
Initial best: f(x) = 8.248647
Iter 1 | Best: 6.729488 | Rate: 1.00 | Evals: 30.0%
Iter 2 | Best: 6.729488 | Curr: 12.798052 | Rate: 0.50 | Evals: 35.0%
Iter 3 | Best: 2.955225 | Rate: 0.67 | Evals: 40.0%
Iter 4 | Best: 1.059837 | Rate: 0.75 | Evals: 45.0%
Iter 5 | Best: 0.147630 | Rate: 0.80 | Evals: 50.0%
Iter 6 | Best: 0.037764 | Rate: 0.83 | Evals: 55.0%
Iter 7 | Best: 0.026664 | Rate: 0.86 | Evals: 60.0%
Iter 8 | Best: 0.000057 | Rate: 0.88 | Evals: 65.0%
Iter 9 | Best: 0.000005 | Rate: 0.89 | Evals: 70.0%
Iter 10 | Best: 0.000005 | Rate: 0.90 | Evals: 75.0%
Iter 11 | Best: 0.000005 | Rate: 0.91 | Evals: 80.0%
Iter 12 | Best: 0.000003 | Rate: 0.92 | Evals: 85.0%
Iter 13 | Best: 0.000003 | Rate: 0.92 | Evals: 90.0%
Iter 14 | Best: 0.000003 | Curr: 0.000003 | Rate: 0.86 | Evals: 95.0%
Iter 15 | Best: 0.000003 | Curr: 0.000004 | Rate: 0.80 | Evals: 100.0%
Best point found: [-0.00045042 0.00161644]
Best value: 0.000003
Total evaluations: 20
Sequential iterations: 15
Success: True
Message: Optimization terminated: maximum evaluations (20) reached
Current function value: 0.000003
Iterations: 15
Function evaluations: 20# Create optimizer
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
n_initial=5,
max_iter=20,
verbose=True,
acquisition_optimizer='tricands',
)
# Run optimization
result = opt.optimize()
print(f"\nBest point found: {result.x}")
print(f"Best value: {result.fun:.6f}")
print(f"Total evaluations: {result.nfev}")
print(f"Sequential iterations: {result.nit}")
print(f"Success: {result.success}")
print(f"Message: {result.message}")TensorBoard logging disabled
Initial best: f(x) = 4.183181
Iter 1 | Best: 4.183181 | Curr: 4.669404 | Rate: 0.00 | Evals: 30.0%
Iter 2 | Best: 4.183181 | Curr: 4.903709 | Rate: 0.00 | Evals: 35.0%
Iter 3 | Best: 1.218197 | Rate: 0.33 | Evals: 40.0%
Iter 4 | Best: 1.218197 | Curr: 2.837087 | Rate: 0.25 | Evals: 45.0%
Iter 5 | Best: 1.145802 | Rate: 0.40 | Evals: 50.0%
Iter 6 | Best: 0.327620 | Rate: 0.50 | Evals: 55.0%
Iter 7 | Best: 0.327620 | Curr: 0.447909 | Rate: 0.43 | Evals: 60.0%
Iter 8 | Best: 0.112760 | Rate: 0.50 | Evals: 65.0%
Iter 9 | Best: 0.112760 | Curr: 0.194964 | Rate: 0.44 | Evals: 70.0%
Iter 10 | Best: 0.112760 | Curr: 0.176636 | Rate: 0.40 | Evals: 75.0%
Iter 11 | Best: 0.112760 | Curr: 0.133378 | Rate: 0.36 | Evals: 80.0%
Iter 12 | Best: 0.112760 | Curr: 0.130906 | Rate: 0.33 | Evals: 85.0%
Iter 13 | Best: 0.112760 | Curr: 0.117525 | Rate: 0.31 | Evals: 90.0%
Iter 14 | Best: 0.112760 | Curr: 0.113271 | Rate: 0.29 | Evals: 95.0%
Iter 15 | Best: 0.111987 | Rate: 0.33 | Evals: 100.0%
Best point found: [0.27742002 0.18715073]
Best value: 0.111987
Total evaluations: 20
Sequential iterations: 15
Success: True
Message: Optimization terminated: maximum evaluations (20) reached
Current function value: 0.111987
Iterations: 15
Function evaluations: 20These methods handle the creation and validation of the initial design of experiments.
get_initial_design()Purpose: Generate or process initial design points
Used by: optimize() method at the start
Calls: _generate_initial_design() if X0 is None
This method handles three scenarios: 1. Generate LHS design when X0=None 2. Include starting point x0 if provided 3. Transform user-provided X0
# Example 1: Generate default LHS design
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
n_initial=10,
seed=42
)
X0 = opt.get_initial_design()
print(f"Generated LHS design shape: {X0.shape}")
print(f"First 3 points:\n{X0[:3]}")
# Example 2: With starting point x0
opt_x0 = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
n_initial=10,
x0=[0.0, 0.0],
seed=42
)
X0_with_x0 = opt_x0.get_initial_design()
print(f"\nDesign with x0, first point: {X0_with_x0[0]}")
# Example 3: Provide custom initial design
X0_custom = np.array([[0, 0], [1, 1], [2, 2]])
X0_processed = opt.get_initial_design(X0_custom)
print(f"\nCustom design shape: {X0_processed.shape}")Generated LHS design shape: (10, 2)
First 3 points:
[[-2.77395605 1.56112156]
[ 4.14140208 -1.69736803]
[-4.09417735 3.02437765]]
Design with x0, first point: [0. 0.]
Custom design shape: (3, 2)_curate_initial_design()Purpose: Remove duplicates and ensure sufficient unique points
Used by: optimize() after get_initial_design()
Handles: Duplicate removal, point generation, repetition for noisy functions
# Example 1: Remove duplicates
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
n_initial=10,
seed=42
)
X0_with_dups = np.array([[1, 2], [1, 2], [3, 4], [3, 4], [5, 6]])
X0_curated = opt._curate_initial_design(X0_with_dups)
print(f"Original points: {len(X0_with_dups)}")
print(f"After removing duplicates: {len(X0_curated)}")
# Example 2: With repeats for noisy functions
opt_repeat = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
n_initial=5,
repeats_initial=3, # Repeat each point 3 times
seed=42
)
X0 = np.array([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10]])
X0_repeated = opt_repeat._curate_initial_design(X0)
print(f"\nOriginal points: {len(X0)}")
print(f"After repeating (3x): {len(X0_repeated)}")Original points: 5
After removing duplicates: 10
Original points: 5
After repeating (3x): 15_rm_NA_values()Purpose: Remove NaN/inf values from initial design evaluations
Used by: optimize() after evaluating initial design
Returns: Cleaned arrays and original count
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
n_initial=10,
seed=42
)
# Simulate initial design with some NaN values
X0 = np.array([[1, 2], [3, 4], [5, 6]])
y0 = np.array([5.0, np.nan, np.inf])
X0_clean, y0_clean, n_eval = opt._rm_NA_values(X0, y0)
print(f"Original evaluations: {n_eval}")
print(f"Valid points remaining: {X0_clean.shape[0]}")
print(f"Clean X0:\n{X0_clean}")
print(f"Clean y0: {y0_clean}")Original evaluations: 3
Valid points remaining: 1
Clean X0:
[[1 2]]
Clean y0: [5.]_check_size_initial_design()Purpose: Validate sufficient points for surrogate fitting
Used by: optimize() after handling NaN values
Raises: ValueError if insufficient points
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
n_initial=10,
seed=42
)
# Example 1: Sufficient points - no error
y0_sufficient = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
try:
opt._check_size_initial_design(y0_sufficient, n_evaluated=10)
print("✓ Sufficient points - validation passed")
except ValueError as e:
print(f"✗ Error: {e}")
# Example 2: Insufficient points - raises error
y0_insufficient = np.array([1.0]) # Only 1 point, need at least 3 for 2D
try:
opt._check_size_initial_design(y0_insufficient, n_evaluated=10)
print("✓ Validation passed")
except ValueError as e:
print(f"✗ Expected error: {e}")✓ Sufficient points - validation passed
✗ Expected error: Insufficient valid initial design points: only 1 finite value(s) out of 10 evaluated. Need at least 3 points to fit surrogate model. Please check your objective function or increase n_initial._get_best_xy_initial_design()Purpose: Determine and store the best point from initial design
Used by: optimize() after initial design evaluation
Updates: self.best_x_ and self.best_y_ attributes
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
n_initial=5,
verbose=True,
seed=42
)
# Simulate initial design (normally done in optimize())
opt.X_ = np.array([[1, 2], [0, 0], [2, 1]])
opt.y_ = np.array([5.0, 0.0, 5.0])
opt._get_best_xy_initial_design()
print(f"\nBest x from initial design: {opt.best_x_}")
print(f"Best y from initial design: {opt.best_y_}")TensorBoard logging disabled
Initial best: f(x) = 0.000000
Best x from initial design: [0 0]
Best y from initial design: 0.0These methods handle surrogate model operations during the optimization loop.
_fit_surrogate()Purpose: Fit surrogate model to data
Used by: optimize() in main loop
Calls: _selection_dispatcher() if max_surrogate_points exceeded
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
max_surrogate_points=10,
seed=42
)
# Generate some training data
X = np.random.rand(50, 2) * 10 - 5 # 50 points in [-5, 5]
y = np.sum(X**2, axis=1)
# Fit surrogate (will select 10 best points)
opt._fit_surrogate(X, y)
print(f"Surrogate fitted successfully!")
print(f"Surrogate model: {type(opt.surrogate).__name__}")Surrogate fitted successfully!
Surrogate model: GaussianProcessRegressor_predict_with_uncertainty()Purpose: Predict with uncertainty estimates, handling surrogates without return_std
Used by: _acquisition_function() and plot_surrogate()
Returns: Predictions and standard deviations
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
surrogate=GaussianProcessRegressor(),
seed=42
)
# Train surrogate
X_train = np.array([[0, 0], [1, 1], [2, 2]])
y_train = np.array([0, 2, 8])
opt._fit_surrogate(X_train, y_train)
# Predict on new points
X_test = np.array([[1.5, 1.5], [3.0, 3.0]])
preds, stds = opt._predict_with_uncertainty(X_test)
print(f"Test points:\n{X_test}")
print(f"Predictions: {preds}")
print(f"Uncertainties: {stds}")Test points:
[[1.5 1.5]
[3. 3. ]]
Predictions: [5.66013019 3.0829763 ]
Uncertainties: [0.31263595 0.92017596]_acquisition_function()Purpose: Compute acquisition function value
Used by: suggest_next_infill_point() for optimization
Calls: _predict_with_uncertainty()
Supports: Expected Improvement (EI), Probability of Improvement (PI), Mean prediction
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
surrogate=GaussianProcessRegressor(),
acquisition='ei',
seed=42
)
# Setup
X_train = np.array([[0, 0], [1, 1], [2, 2]])
y_train = np.array([0, 2, 8])
opt._fit_surrogate(X_train, y_train)
opt.y_ = y_train # Needed for acquisition function
# Evaluate acquisition function
x_eval = np.array([1.5, 1.5])
acq_value = opt._acquisition_function(x_eval)
print(f"Point to evaluate: {x_eval}")
print(f"Acquisition function value (EI): {acq_value:.6f}")
print(f"(Lower is better for minimization)")Point to evaluate: [1.5 1.5]
Acquisition function value (EI): -0.000000
(Lower is better for minimization)suggest_next_infill_point()Purpose: Suggest next point to evaluate using acquisition function optimization
Used by: optimize() in main loop
Calls: _acquisition_function(), _handle_acquisition_failure() if needed
Handles: Integer/factor rounding, duplicate avoidance
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
surrogate=GaussianProcessRegressor(),
acquisition='ei',
seed=42
)
# Setup
X_train = np.array([[0, 0], [1, 1], [2, 2]])
y_train = np.array([0, 2, 8])
opt._fit_surrogate(X_train, y_train)
opt.X_ = X_train
opt.y_ = y_train
# Suggest next point
x_next = opt.suggest_next_infill_point()
print(f"Next point to evaluate: {x_next}")
print(f"Expected to be between known points or in unexplored regions")Next point to evaluate: [[-1.6288683 1.99062025]]
Expected to be between known points or in unexplored regions_apply_ocba()Purpose: Apply OCBA for noisy functions to determine which points to re-evaluate
Used by: optimize() in main loop when noise=True and ocba_delta > 0
Returns: Points to re-evaluate or None
# Example with OCBA enabled
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
repeats_initial=2, # Enable noise handling
ocba_delta=5,
verbose=True,
seed=42
)
# Simulate optimization state
opt.mean_X = np.array([[1, 2], [0, 0], [2, 1], [1, 1]])
opt.mean_y = np.array([5.0, 0.1, 5.0, 2.0])
opt.var_y = np.array([0.1, 0.05, 0.15, 0.08])
X_ocba = opt._apply_ocba()
if X_ocba is not None:
print(f"\nOCBA returned {X_ocba.shape[0]} points for re-evaluation")
else:
print("\nOCBA: No re-evaluation needed")TensorBoard logging disabled
In _get_ocba():
means: [5. 0.1 5. 2. ]
vars: [0.1 0.05 0.15 0.08]
delta: 5
n_designs: 4
Ratios: [0.18794252 0.81801772 0.28191379 1. ]
Best: 1, Second best: 3
OCBA: Adding 5 re-evaluation(s)
OCBA returned 5 points for re-evaluation_apply_penalty_NA()Purpose: Replace NaN and infinite values with penalty plus random noise
Used by: _handle_NA_new_points() and indirectly by optimize()
Algorithm: penalty = max(finite_y) + 3 * std(finite_y) + noise
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5)],
seed=42
)
# Historical values (used for penalty computation)
y_hist = np.array([1.0, 2.0, 3.0, 5.0])
# New evaluations with NaN/inf
y_new = np.array([4.0, np.nan, np.inf])
y_clean = opt._apply_penalty_NA(y_new, y_history=y_hist)
print(f"Original: {y_new}")
print(f"After penalty: {y_clean}")
print(f"All finite: {np.all(np.isfinite(y_clean))}")
print(f"Penalty values > max(hist): {y_clean[1] > np.max(y_hist)}")Original: [ 4. nan inf]
After penalty: [ 4. 10.10964895 10.18824424]
All finite: True
Penalty values > max(hist): True_remove_nan()Purpose: Remove rows where y contains NaN or inf values
Used by: optimize() after function evaluations
Returns: Cleaned arrays
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5)],
seed=42
)
X = np.array([[1, 2], [3, 4], [5, 6]])
y = np.array([1.0, np.nan, np.inf])
X_clean, y_clean = opt._remove_nan(X, y, stop_on_zero_return=False)
print(f"Original X shape: {X.shape}")
print(f"Clean X shape: {X_clean.shape}")
print(f"Clean X:\n{X_clean}")
print(f"Clean y: {y_clean}")Original X shape: (3, 2)
Clean X shape: (1, 2)
Clean X:
[[1 2]]
Clean y: [1.]_handle_NA_new_points()Purpose: Handle NaN/inf values in new evaluation points during main loop
Used by: optimize() after evaluating new points
Calls: _apply_penalty_NA() and _remove_nan()
Returns: None, None if all evaluations invalid (skip iteration)
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
n_initial=5,
verbose=True,
seed=42
)
# Simulate optimization state
opt.y_ = np.array([1.0, 2.0, 3.0]) # Historical values
opt.n_iter_ = 1
# Case 1: Some valid values
print("Case 1: Some valid evaluations")
x_next = np.array([[1, 2], [3, 4], [5, 6]])
y_next = np.array([5.0, np.nan, 10.0])
x_clean, y_clean = opt._handle_NA_new_points(x_next, y_next)
print(f"Valid points remaining: {x_clean.shape[0] if x_clean is not None else 0}")
# Case 2: All invalid - should return None
print("\nCase 2: All invalid evaluations")
x_all_bad = np.array([[1, 2], [3, 4]])
y_all_bad = np.array([np.nan, np.inf])
x_clean, y_clean = opt._handle_NA_new_points(x_all_bad, y_all_bad)
print(f"Result: {'None (skip iteration)' if x_clean is None else 'Valid points'}")TensorBoard logging disabled
Case 1: Some valid evaluations
Warning: Removed 1 sample(s) with NaN/inf values
Valid points remaining: 2
Case 2: All invalid evaluations
Warning: All objective function values are NaN or inf. Returning empty arrays.
Warning: All new evaluations were NaN/inf, skipping iteration 1
Result: None (skip iteration)_update_best_main_loop()Purpose: Update best solution found during main optimization loop
Used by: optimize() after each iteration
Updates: self.best_x_ and self.best_y_ if improvement found
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
n_initial=5,
verbose=True,
seed=42
)
# Simulate optimization state
opt.n_iter_ = 1
opt.best_x_ = np.array([1.0, 1.0])
opt.best_y_ = 2.0
# Case 1: New best found
print("Case 1: New best found")
x_new = np.array([[0.1, 0.1], [0.5, 0.5]])
y_new = np.array([0.02, 0.5])
opt._update_best_main_loop(x_new, y_new)
print(f"Updated best_y: {opt.best_y_}\n")
# Case 2: No improvement
print("Case 2: No improvement")
opt.n_iter_ = 2
x_no_improve = np.array([[1.5, 1.5]])
y_no_improve = np.array([4.5])
opt._update_best_main_loop(x_no_improve, y_no_improve)
print(f"Best_y unchanged: {opt.best_y_}")TensorBoard logging disabled
Case 1: New best found
Iter 1 | Best: 0.020000 | Rate: 0.00 | Evals: 0.0%
Updated best_y: 0.02
Case 2: No improvement
Iter 2 | Best: 0.020000 | Curr: 4.500000 | Rate: 0.00 | Evals: 0.0%
Best_y unchanged: 0.02_determine_termination()Purpose: Determine termination reason for optimization
Used by: optimize() at the end
Checks: Max iterations, time limit, or successful completion
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
max_iter=20,
max_time=10.0,
seed=42
)
# Case 1: Maximum evaluations reached
print("Case 1: Maximum evaluations reached")
opt.y_ = np.zeros(20)
start_time = time.time()
msg = opt._determine_termination(start_time)
print(f"Message: {msg}\n")
# Case 2: Time limit exceeded (simulated)
print("Case 2: Time limit exceeded")
opt.y_ = np.zeros(10)
start_time = time.time() - 700 # 11.67 minutes ago
msg = opt._determine_termination(start_time)
print(f"Message: {msg}\n")
# Case 3: Successful completion
print("Case 3: Successful completion")
opt.y_ = np.zeros(10)
start_time = time.time()
msg = opt._determine_termination(start_time)
print(f"Message: {msg}")Case 1: Maximum evaluations reached
Message: Optimization terminated: maximum evaluations (20) reached
Case 2: Time limit exceeded
Message: Optimization terminated: time limit (10.00 min) reached
Case 3: Successful completion
Message: Optimization finished successfullyselect_new()Purpose: Select rows from A that are not in X (avoid duplicate evaluations)
Used by: suggest_next_infill_point() to ensure new points are different from evaluated points
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5)],
seed=42
)
A = np.array([[1, 2], [3, 4], [5, 6]])
X = np.array([[3, 4], [7, 8]])
new_A, is_new = opt.select_new(A, X)
print(f"Candidate points A:\n{A}")
print(f"\nKnown points X:\n{X}")
print(f"\nNew points from A:\n{new_A}")
print(f"\nIs new mask: {is_new}")Candidate points A:
[[1 2]
[3 4]
[5 6]]
Known points X:
[[3 4]
[7 8]]
New points from A:
[[1 2]
[5 6]]
Is new mask: [ True False True]_repair_non_numeric()Purpose: Round non-numeric values (int, factor) to integers
Used by: Various methods to ensure integer/factor variables have valid values
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
var_type=['int', 'float'],
seed=42
)
X = np.array([[1.2, 2.5], [3.7, 4.1], [5.9, 6.8]])
X_repaired = opt._repair_non_numeric(X.copy(), opt.var_type)
print(f"Original X:\n{X}")
print(f"\nRepaired X (first column rounded to int):\n{X_repaired}")Original X:
[[1.2 2.5]
[3.7 4.1]
[5.9 6.8]]
Repaired X (first column rounded to int):
[[1. 2.5]
[4. 4.1]
[6. 6.8]]_map_to_factor_values()Purpose: Map internal integer values to original factor strings
Used by: optimize() when preparing results for user
Handles: Factor (categorical) variables
opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), ('red', 'green', 'blue')],
var_type=['float', 'factor'],
seed=42
)
# Internal representation (integers for factors)
X_internal = np.array([[1.0, 0], [2.0, 1], [3.0, 2]])
X_mapped = opt._map_to_factor_values(X_internal)
print(f"Internal representation:\n{X_internal}")
print(f"\nMapped to factor values:\n{X_mapped}")Internal representation:
[[1. 0.]
[2. 1.]
[3. 2.]]
Mapped to factor values:
[[1.0 'red']
[2.0 'green']
[3.0 'blue']]opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
max_iter=15,
seed=42,
verbose=True
)
# Provide custom initial design
X0 = np.array([[0, 0], [1, 1], [2, 2], [-1, -1], [-2, -2]])
result = opt.optimize(X0=X0)
print(f"\nBest point: {result.x}")
print(f"Best value: {result.fun:.6f}")TensorBoard logging disabled
Note: Initial design size (5) is smaller than requested (10) due to NaN/inf values
Initial best: f(x) = 0.000000
Iter 1 | Best: 0.000000 | Curr: 0.000000 | Rate: 0.00 | Evals: 40.0%
Iter 2 | Best: 0.000000 | Curr: 0.000000 | Rate: 0.00 | Evals: 46.7%
Optimizer candidate 1/3 was duplicate/invalid.
Optimizer candidate 2/3 was duplicate/invalid.
Iter 3 | Best: 0.000000 | Curr: 0.000000 | Rate: 0.00 | Evals: 53.3%
Optimizer candidate 1/3 was duplicate/invalid.
Optimizer candidate 2/3 was duplicate/invalid.
Iter 4 | Best: 0.000000 | Curr: 0.000000 | Rate: 0.00 | Evals: 60.0%
Iter 5 | Best: 0.000000 | Curr: 0.000000 | Rate: 0.00 | Evals: 66.7%
Iter 6 | Best: 0.000000 | Curr: 0.000000 | Rate: 0.00 | Evals: 73.3%
Iter 7 | Best: 0.000000 | Curr: 0.000000 | Rate: 0.00 | Evals: 80.0%
Iter 8 | Best: 0.000000 | Curr: 0.000000 | Rate: 0.00 | Evals: 86.7%
Iter 9 | Best: 0.000000 | Curr: 0.000000 | Rate: 0.00 | Evals: 93.3%
Optimizer candidate 1/3 was duplicate/invalid.
Iter 10 | Best: 0.000000 | Curr: 0.000000 | Rate: 0.00 | Evals: 100.0%
Best point: [0 0]
Best value: 0.000000opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
max_iter=15,
n_initial=5,
x0=[0.5, 0.5], # Starting point near optimum
seed=42,
verbose=True
)
result = opt.optimize()
print(f"\nBest point: {result.x}")
print(f"Best value: {result.fun:.6f}")Starting point x0 validated and processed successfully.
Original scale: [0.5 0.5]
Internal scale: [0.5 0.5]
TensorBoard logging disabled
Including 1 starting points from x0 in initial design.
Initial best: f(x) = 0.500000
Iter 1 | Best: 0.500000 | Curr: 0.691045 | Rate: 0.00 | Evals: 40.0%
Iter 2 | Best: 0.149342 | Rate: 0.50 | Evals: 46.7%
Iter 3 | Best: 0.015439 | Rate: 0.67 | Evals: 53.3%
Iter 4 | Best: 0.000532 | Rate: 0.75 | Evals: 60.0%
Iter 5 | Best: 0.000072 | Rate: 0.80 | Evals: 66.7%
Iter 6 | Best: 0.000004 | Rate: 0.83 | Evals: 73.3%
Iter 7 | Best: 0.000003 | Rate: 0.86 | Evals: 80.0%
Iter 8 | Best: 0.000003 | Rate: 0.88 | Evals: 86.7%
Iter 9 | Best: 0.000003 | Rate: 0.89 | Evals: 93.3%
Iter 10 | Best: 0.000003 | Rate: 0.90 | Evals: 100.0%
Best point: [0.00168783 0.00020213]
Best value: 0.000003opt = SpotOptim(
fun=sphere,
bounds=[(-5, 5), (-5, 5)],
var_type=['int', 'int'],
max_iter=20,
n_initial=10,
seed=42,
verbose=True
)
result = opt.optimize()
print(f"\nBest point: {result.x}")
print(f"Best value: {result.fun:.6f}")
print(f"Point has integers: {np.all(result.x == np.round(result.x))}")TensorBoard logging disabled
Initial best: f(x) = 4.000000
Optimizer candidate 1/3 was duplicate/invalid.
Optimizer candidate 2/3 was duplicate/invalid.
Optimizer candidate 3/3 was duplicate/invalid.
Fallback attempt 1/10: Using fallback strategy
Acquisition failure: Using random space-filling design as fallback.
Iter 1 | Best: 4.000000 | Curr: 29.000000 | Rate: 0.00 | Evals: 55.0%
Optimizer candidate 1/3 was duplicate/invalid.
Optimizer candidate 2/3 was duplicate/invalid.
Optimizer candidate 3/3 was duplicate/invalid.
Fallback attempt 1/10: Using fallback strategy
Acquisition failure: Using random space-filling design as fallback.
Iter 2 | Best: 1.000000 | Rate: 0.50 | Evals: 60.0%
Iter 3 | Best: 0.000000 | Rate: 0.67 | Evals: 65.0%
Optimizer candidate 1/3 was duplicate/invalid.
Optimizer candidate 2/3 was duplicate/invalid.
Optimizer candidate 3/3 was duplicate/invalid.
Fallback attempt 1/10: Using fallback strategy
Acquisition failure: Using random space-filling design as fallback.
Iter 4 | Best: 0.000000 | Curr: 25.000000 | Rate: 0.50 | Evals: 70.0%
Optimizer candidate 1/3 was duplicate/invalid.
Optimizer candidate 2/3 was duplicate/invalid.
Optimizer candidate 3/3 was duplicate/invalid.
Fallback attempt 1/10: Using fallback strategy
Acquisition failure: Using random space-filling design as fallback.
Iter 5 | Best: 0.000000 | Curr: 9.000000 | Rate: 0.40 | Evals: 75.0%
Optimizer candidate 1/3 was duplicate/invalid.
Optimizer candidate 2/3 was duplicate/invalid.
Optimizer candidate 3/3 was duplicate/invalid.
Fallback attempt 1/10: Using fallback strategy
Acquisition failure: Using random space-filling design as fallback.
Iter 6 | Best: 0.000000 | Curr: 5.000000 | Rate: 0.33 | Evals: 80.0%
Optimizer candidate 1/3 was duplicate/invalid.
Optimizer candidate 2/3 was duplicate/invalid.
Optimizer candidate 3/3 was duplicate/invalid.
Fallback attempt 1/10: Using fallback strategy
Acquisition failure: Using random space-filling design as fallback.
Iter 7 | Best: 0.000000 | Curr: 13.000000 | Rate: 0.29 | Evals: 85.0%
Optimizer candidate 1/3 was duplicate/invalid.
Optimizer candidate 2/3 was duplicate/invalid.
Optimizer candidate 3/3 was duplicate/invalid.
Fallback attempt 1/10: Using fallback strategy
Acquisition failure: Using random space-filling design as fallback.
Iter 8 | Best: 0.000000 | Curr: 13.000000 | Rate: 0.25 | Evals: 90.0%
Optimizer candidate 1/3 was duplicate/invalid.
Optimizer candidate 2/3 was duplicate/invalid.
Optimizer candidate 3/3 was duplicate/invalid.
Fallback attempt 1/10: Using fallback strategy
Acquisition failure: Using random space-filling design as fallback.
Iter 9 | Best: 0.000000 | Curr: 10.000000 | Rate: 0.22 | Evals: 95.0%
Optimizer candidate 1/3 was duplicate/invalid.
Optimizer candidate 2/3 was duplicate/invalid.
Optimizer candidate 3/3 was duplicate/invalid.
Fallback attempt 1/10: Using fallback strategy
Acquisition failure: Using random space-filling design as fallback.
Iter 10 | Best: 0.000000 | Curr: 8.000000 | Rate: 0.20 | Evals: 100.0%
Best point: [-0. -0.]
Best value: 0.000000
Point has integers: True# Noisy sphere function
def noisy_sphere(X):
X = np.atleast_2d(X)
return np.sum(X**2, axis=1) + np.random.normal(0, 0.1, X.shape[0])
opt = SpotOptim(
fun=noisy_sphere,
bounds=[(-5, 5), (-5, 5)],
max_iter=25,
n_initial=10,
repeats_initial=2, # Enable noise handling
ocba_delta=5, # Enable OCBA with 5 re-evaluations
seed=42,
verbose=True
)
result = opt.optimize()
print(f"\nBest point: {result.x}")
print(f"Best value: {result.fun:.6f}")
print(f"Note: OCBA re-evaluated promising points to reduce uncertainty")TensorBoard logging disabled
Initial best: f(x) = 2.319523, mean best: f(x) = 2.385877
In _get_ocba():
means: [25.8857338 11.32118531 10.14985518 2.38587731 20.7190555 21.36600973
16.36904227 14.1099758 18.36970272 20.14080533]
vars: [6.73858271e-13 1.33640624e-08 1.00799409e-03 4.40284305e-03
2.56053422e-03 6.35744853e-04 1.16126758e-02 3.37927306e-03
1.64745915e-03 1.91555606e-03]
delta: 5
n_designs: 10
Ratios: [7.29707371e-11 1.00099067e-05 1.00000000e+00 3.61962598e+00
4.55581054e-01 1.05534624e-01 3.55166445e+00 1.47019506e+00
3.85623766e-01 3.63385525e-01]
Best: 3, Second best: 2
OCBA: Adding 5 re-evaluation(s)
Iter 1 | Best: 2.319523 | Curr: 2.393985 | Rate: 0.00 | Evals: 104.0% | Mean Curr: 9.049830
Best point: [-1.55458479 -0.06381726]
Best value: 2.319523
Note: OCBA re-evaluated promising points to reduce uncertainty# Function that sometimes returns NaN
def sometimes_nan(X):
X = np.atleast_2d(X)
y = np.sum(X**2, axis=1)
# Return NaN for large values (penalty will be applied)
y[y > 100] = np.nan
return y
opt = SpotOptim(
fun=sometimes_nan,
bounds=[(-10, 10), (-10, 10)],
max_iter=20,
n_initial=10,
seed=42,
verbose=True
)
result = opt.optimize()
print(f"\nBest point: {result.x}")
print(f"Best value: {result.fun:.6f}")
print(f"Optimization succeeded despite NaN values: {result.success}")TensorBoard logging disabled
Warning: 1 initial design point(s) returned NaN/inf and will be ignored (reduced from 10 to 9 points)
Note: Initial design size (9) is smaller than requested (10) due to NaN/inf values
Initial best: f(x) = 9.683226
Iter 1 | Best: 8.496828 | Rate: 1.00 | Evals: 50.0%
Iter 2 | Best: 4.452331 | Rate: 1.00 | Evals: 55.0%
Iter 3 | Best: 0.175858 | Rate: 1.00 | Evals: 60.0%
Iter 4 | Best: 0.033204 | Rate: 1.00 | Evals: 65.0%
Iter 5 | Best: 0.000265 | Rate: 1.00 | Evals: 70.0%
Iter 6 | Best: 0.000265 | Curr: 0.000388 | Rate: 0.83 | Evals: 75.0%
Iter 7 | Best: 0.000083 | Rate: 0.86 | Evals: 80.0%
Iter 8 | Best: 0.000005 | Rate: 0.88 | Evals: 85.0%
Iter 9 | Best: 0.000002 | Rate: 0.89 | Evals: 90.0%
Iter 10 | Best: 0.000001 | Rate: 0.90 | Evals: 95.0%
Iter 11 | Best: 0.000001 | Curr: 0.000001 | Rate: 0.82 | Evals: 100.0%
Best point: [-0.00070328 -0.00046471]
Best value: 0.000001
Optimization succeeded despite NaN values: TrueHere’s how all the methods relate to each other in the optimization workflow:
optimize()
│
├─── Initial Design Phase
│ ├── get_initial_design()
│ │ └── _generate_initial_design() [if X0 is None]
│ ├── _curate_initial_design()
│ ├── _evaluate_function()
│ ├── _rm_NA_values()
│ ├── _check_size_initial_design()
│ └── _get_best_xy_initial_design()
│
├─── Main Optimization Loop (while not terminated)
│ │
│ ├── _fit_surrogate()
│ │ └── _selection_dispatcher() [if max_surrogate_points exceeded]
│ │
│ ├── _apply_ocba() [if noise=True and ocba_delta > 0]
│ │
│ ├── suggest_next_infill_point()
│ │ ├── _acquisition_function()
│ │ │ └── _predict_with_uncertainty()
│ │ ├── _repair_non_numeric()
│ │ ├── select_new()
│ │ └── _handle_acquisition_failure() [if needed]
│ │
│ ├── _evaluate_function()
│ │
│ ├── _handle_NA_new_points()
│ │ ├── _apply_penalty_NA()
│ │ └── _remove_nan()
│ │
│ └── _update_best_main_loop()
│
└─── Termination Phase
├── _determine_termination()
└── _map_to_factor_values() [for results]This notebook demonstrated all major methods in SpotOptim with executable examples:
Core Flow:
Key Features:
All examples can be run independently and demonstrate the modular design of SpotOptim!