Optuna is an open-source Python library for automatic hyperparameter tuning of machine learning models. It was first introduced in 2018 by Preferred Networks, a Japanese startup that works on practical applications of deep learning in various fields. The beta version of Optuna was released at the end of the year, with the subsequent first major stable release announced in January 2020.
Hyperparameter optimization
Hyperparameter optimization involves finding the optimal value of non-trainable parameters, defined by the user. Examples of hyperparameters are learning rate, number of layers or neurons, regularization strength, and tree depth. However, they strongly depend on the specific algorithm (e.g., classification, regression, clustering, etc.). Hyperparameter optimization can be especially relevant when dealing with large-scale problems or limited resources, as it improves model accuracy, reduces overfitting, and decreases training time. However, when the hyperparameter space increases, it may become computationally expensive. Hence, there are methods (e.g., grid search, random search, or bayesian optimization) that considerably simplify this process.
Features Optuna is designed to optimize the model hyperparameters by searching large spaces and discarding combinations that show no significant improvements in the model. Moreover, it can parallelize the hyperparameter search over multiple threads or processes. Optuna works with a high degree of modularity, allowing the definition of different configurations for searching hyperparameters. It is possible to choose which hyperparameters to optimize and how to optimize them. Additionally, it permits parameter customization at runtime, meaning that the values to explore for the hyperparameters (i.e., the search space) can be defined while writing the code, rather than being defined in advance.
Sampling Optuna exploits well-established algorithms to perform hyperparameter optimization, progressively reducing the search space in light of objective values. Examples are gaussian-process-based algorithms (i.e., a gaussian process to model the objective function), tree-structured parzen estimator (TPE) (i.e., a model-based optimization method that estimates the objective function and selects the best hyperparameters), and random search (i.e., a basic optimization approach used for benchmarking).
Early stopping
Optuna includes a pruning feature to stop trials early when the results show no significant improvement in the model. This allows for the prevention of unnecessary computation, and it is used for models with long training times in order to save time and computational resources. Specifically, Optuna exploits techniques such as median and threshold pruning.
Scalability Optuna is designed to scale with distributed computing resources, supporting parallel execution. This feature allows users to run optimization trials across multiple processors or machines.
Integration with third-part libraries Optuna integrates with various machine-learning (ML) libraries and frameworks:
Catboost Dask Fast.ai Keras LightGBM MLflow PyTorch PyTorch Ignite PyTorch Lightning TensorBoard TensorFlow tf.keras Weights & Biases XGBoost skrub Moreover, Optuna offers a real-time dashboard that allows a user to monitor, through graphs and tables, the optimization history and the hyperparameter importance.
Applications Optuna was designed to be framework-agnostic so that it can be used with any machine-learning (ML) or deep-learning (DL) framework.
Machine learning
Optuna can be used to optimize hyperparameters of ML models. Examples are:
Random forest: number of trees, maximum depth, and minimum samples per leaf. Gradient boosting machines (GBM): learning rate, number of estimators, and maximum depth. Support vector machines (SVM): regularization parameter (C), kernel type (e.g., linear, radial basis function), and gamma (gamma). K-nearest neighbors (KNN): number of neighbors (k), distance metrics (e.g., Euclidean or Manhattan), and weight function. Linear and logistic regression: alpha in Ridge Regression or C in Logistic Regression. Naive Bayes: smoothing coefficients.
Deep learning In the context of deep learning, Optuna can be exploited during the training of neural networks (NN) to optimize learning rate, batch size, and the number of hidden layers. For example, it can be used for:
Convolutional neural networks (CNNs), for image classification, object detection, and semantic-segmentation tasks. Recurrent neural networks (RNNs), for sequence-based tasks such as time-series forecasting and natural language processing. Transformers, for NLP tasks such as text classification, sentiment analysis, and machine translation.
Domains Optuna has found applications in various research studies and industry implementations across different applicative domains.
Healthcare In healthcare, Optuna is currently utilized for medical image analysis to optimize DL models for tumor detection, disease identification, multi-organ semantic segmentation, and radiological image analysis. It can also be implemented for disease prediction, in particular for the improvement in the accuracy of predictive models for disease diagnosis and treatment outcomes. Its application can also be found in genomics to predict genetic diseases and to identify genetic variations.
Finance In finance, Optuna is used to optimize models for algorithmic trading. It allows for predicting market movements due to its ability to handle wide parameter ranges and complex model structures. In particular, it is exploited for financial risk analysis and forecasting. Topics addressed are credit, market, and operational risks.
Autonomous systems Optuna is used for real-time applications in autonomous systems for robotics, supporting decision making in dynamic environments. It is also used in the context of self-driving cars to optimize the model to navigate safely in complex environments. For example, Optuna can be used in scenarios where there is the need to evaluate the rate and the severity of accidents or to address the issue of network intrusion attacks due to potential vulnerabilities.
… excerpt ends here. Continue reading the full article.
