1  Introduction to SpotOptim

2 Examples

Let us consider the problem of minimizing the Rosenbrock function. This function (and its respective derivatives) is implemented in rosen (and rosen_der, rosen_hess) in the scipy.optimize module.

The function is usually evaluated on the hypercube \(x_i \in [-5, 10]\), for all \(i = 1, \ldots, d\), although it may be restricted to the hypercube $x_i , for all \(i = 1, \ldots, d\), see https://www.sfu.ca/~ssurjano/rosen.html.

A simple application of the Nelder-Mead method is:

from scipy.optimize import minimize, rosen, rosen_der
x0 = [1.3, 0.7, 0.8, 1.9, 1.2]
res = minimize(rosen, x0, method='Nelder-Mead', tol=1e-6, 
        bounds = [(-2.048, 2.048)] * 5)
res.x
array([1.00000002, 1.00000002, 1.00000007, 1.00000015, 1.00000028])

Now using the BFGS algorithm, using the first derivative and a few options:

res = minimize(rosen, x0, method='BFGS', jac=rosen_der,
               options={'gtol': 1e-6, 'disp': True})
res.x
print(res.message)
Optimization terminated successfully.
         Current function value: 0.000000
         Iterations: 26
         Function evaluations: 31
         Gradient evaluations: 31
Optimization terminated successfully.

Now, let’s see how to solve the same problem using SpotOptim, which uses a Surrogate-Model Based Optimization (SMBO) approach. Unlike minimize, SpotOptim requires bounds as it samples the search space globally.

import numpy as np
from spotoptim import SpotOptim
from scipy.optimize import rosen

# SpotOptim expects function input as a 2D array (batch of points)
# So we wrap the rosen function to handle (n_samples, n_dim) input
def rosen_batch(X):
    return np.array([rosen(x) for x in X])

# Define the optimizer
# We use a 5-dimensional problem similar to the example above
# Bounds are required for SpotOptim
optimizer = SpotOptim(
    fun=rosen_batch,
    bounds = [(-2.048, 2.048)] * 5,
    max_iter=50,  # Total budget of function evaluations
    n_initial=10, # Initial random samples
    seed=42
)

# Run optimization
res = optimizer.optimize()
print(res.message)
Optimization terminated: maximum evaluations (50) reached
         Current function value: 2.419963
         Iterations: 40
         Function evaluations: 50
print(f"Best solution found: {res.x}")
print(f"Best objective value: {res.fun}")
print(f"Total evaluations: {res.nfev}")
Best solution found: [0.58093168 0.35010158 0.11108193 0.04121576 0.00122508]
Best objective value: 2.419963301080782
Total evaluations: 50

SpotOptim is particularly useful when the objective function is expensive to evaluate (e.g., simulations, hyperparameter tuning), as it builds a model to intelligently select the next point to evaluate, often finding good solutions with fewer function calls than gradient-free local search methods.

2.1 Complex Constrained Optimization: The Robot Arm

For a more challenging example, let’s consider the robot_arm_hard problem. This is a 10-dimensional problem where a simulated robot arm must reach a target while avoiding obstacles in a maze-like configuration. It features “hard” constraints implemented as severe penalties, creating a rugged landscape that is difficult for local optimizers.

First, let’s try solving it with Scipy’s default local optimizer. Note that we need to wrap the function to return a scalar float, as robot_arm_hard returns an array.

import numpy as np
from scipy.optimize import minimize
from spotoptim.function.so import robot_arm_hard

# Wrapper for Scipy (expects scalar return)
def objective_scipy(x):
    # robot_arm_hard handles constraints internally via penalties
    return float(robot_arm_hard(x)[0])

# Starting point
np.random.seed(42)
x0 = np.random.rand(10)
print(f"Initial cost: {objective_scipy(x0):.4f}")

# Run generic minimization (Nelder-Mead is default for gradient-free)
res_scipy = minimize(objective_scipy, x0, method='Nelder-Mead', tol=1e-4,
            bounds = [(0.0, 1.0)] * 10)

print(f"Scipy Best cost: {res_scipy.fun:.4f}")
print(f"Scipy Success: {res_scipy.success}")
Initial cost: 54.3293
Scipy Best cost: 11.0496
Scipy Success: False

The local optimizer often gets stuck in local optima created by the obstacles (penalties), failing to find a path to the target.

Now let’s use SpotOptim. We don’t need a wrapper because SpotOptim natively supports the batch-vectorized output of robot_arm_hard. We simply define the bounds for the 10 joint angles.

from spotoptim import SpotOptim

# Define bounds: 10 angles normalized to [0, 1]
bounds = [(0.0, 1.0)] * 10

optimizer = SpotOptim(
    fun=robot_arm_hard,  # Native array-based function
    bounds=bounds,
    max_iter=50,         # Budget
    n_initial=20,        # More exploration for complex space
    seed=42,
    max_surrogate_points=30
)

res_spot = optimizer.optimize()

print(f"SpotOptim Best cost: {res_spot.fun:.4f}")
SpotOptim Best cost: 18.7389

SpotOptim’s use of a surrogate model and global acquisition function allows it to better explore the landscape and often jump out of the local traps that catch the local optimizer, finding a significantly better configuration for the robot arm.

2.2 Jupyter Notebook

Note