Hyperparameter Tuning Cookbook
A guide for scikit-learn, PyTorch, river, and spotPython
Preface
This document provides a comprehensive guide to hyperparameter tuning using spotPython for scikit-learn, scipy-optimize, River, and PyTorch. The first part introduces fundamental ideas from optimization. The second part discusses numerical issues and introduces spotPython’s surrogate model-based optimization process. The thirs part focuses on hyperparameter tuning. Several case studies are presented, including hyperparameter tuning for sklearn models such as Support Vector Classification, Random Forests, Gradient Boosting (XGB), and K-nearest neighbors (KNN), as well as a Hoeffding Adaptive Tree Regressor from river. The integration of spotPython into the PyTorch and PyTorch Lightning training workflow is also discussed. With a hands-on approach and step-by-step explanations, this cookbook serves as a practical starting point for anyone interested in hyperparameter tuning with Python. Highlights include the interplay between Tensorboard, PyTorch Lightning, spotPython, spotRiver, and River. This publication is under development, with updates available on the corresponding webpage.
The most recent version of this book is available at https://sequential-parameter-optimization.github.io/Hyperparameter-Tuning-Cookbook/
Book Structure
This document is structured in three parts. The first part presents an introduction to optimization. The second part describes numerical methods, and the third part presents hyperparameter tuning.
- The open access book Bartz et al. (2022) provides a comprehensive overview of hyperparameter tuning. It can be downloaded from https://link.springer.com/book/10.1007/978-981-19-5170-1.
The .ipynb notebook (Bartz-Beielstein 2023) is updated regularly and reflects updates and changes in the spotPython package. It can be downloaded from https://github.com/sequential-parameter-optimization/spotPython/blob/main/notebooks/14_spot_ray_hpt_torch_cifar10.ipynb.
Software Used in this Book
scikit-learn is a Python module for machine learning built on top of SciPy and is distributed under the 3-Clause BSD license. The project was started in 2007 by David Cournapeau as a Google Summer of Code project, and since then many volunteers have contributed.
PyTorch is an optimized tensor library for deep learning using GPUs and CPUs. Lightning is a lightweight PyTorch wrapper for high-performance AI research. It allows you to decouple the research from the engineering.
River is a Python library for online machine learning. It is designed to be used in real-world environments, where not all data is available at once, but streaming in.
spotPython (“Sequential Parameter Optimization Toolbox in Python”) is the Python version of the well-known hyperparameter tuner SPOT, which has been developed in the R programming environment for statistical analysis for over a decade. The related open-access book is available here: Hyperparameter Tuning for Machine and Deep Learning with R—A Practical Guide.
spotRiver provides an interface between spotPython and River.
Citation
If this document has been useful to you and you wish to cite it in a scientific publication, please refer to the following paper, which can be found on arXiv: https://arxiv.org/abs/2307.10262.
@ARTICLE{bart23iArXiv,
author = {{Bartz-Beielstein}, Thomas},
title = "{Hyperparameter Tuning Cookbook:
A guide for scikit-learn, PyTorch, river, and spotPython}",
journal = {arXiv e-prints},
keywords = {Computer Science - Machine Learning,
Computer Science - Artificial Intelligence, 90C26, I.2.6, G.1.6},
year = 2023,
month = jul,
eid = {arXiv:2307.10262},
pages = {arXiv:2307.10262},
doi = {10.48550/arXiv.2307.10262},
archivePrefix = {arXiv},
eprint = {2307.10262},
primaryClass = {cs.LG},
adsurl = {https://ui.adsabs.harvard.edu/abs/2023arXiv230710262B},
adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}