Surrogate-Based Adaptive Algorithms for Optimizing Design in Complex Expensive Black-Box Systems

Optimization of complex systems and processes is critical across various fields such as engineering, finance, and healthcare, where it significantly enhances system performance, reduces costs, and ensures overall success. Specifically, global optimization of an unknown, complex, and expensive objective function, commonly referred to as a ``black-box function'', poses a significant challenge. In a black-box system, the internal mechanisms are not visible or understood; only the inputs and outputs can be observed. This lack of transparency complicates the optimization process because it relies on outcomes without an understanding of the underlying processes. Traditional optimization methods often fall short in these scenarios, where the characteristics of the objective function are unknown. Consequently, evaluating these systems typically requires numerous costly simulations or experiments, making the processes both time-consuming and computationally demanding. Surrogate Optimization (SO) is a prevalent and promising technique for black-box optimization which employs a low-cost surrogate model to guide the optimization process toward the global optimum. However, SO faces significant challenges, particularly in managing the exploration-exploitation trade-off. This trade-off requires the surrogate model to balance between exploring new regions of the function space to uncover potentially superior solutions (exploration) and exploiting regions that are currently believed to be promising based on the surrogate's predictions (exploitation). Moreover, surrogate models often struggle to accurately represent the behavior of the function in high-dimensional spaces, making optimization more difficult and computationally intensive. These challenges are amplified in batch evaluation settings, where black-box function evaluations occur in batches rather than sequentially. This setup adds complexity to balancing exploration and exploitation, as decisions must consider multiple function evaluations simultaneously. In batch SO, the challenge of promoting diversity while pursuing optimal solutions is paramount, hindering the discovery of promising solutions and reducing efficiency. Addressing these challenges requires innovative strategies to effectively manage the exploration-exploitation trade-off and improve the performance of SO in complex optimization scenarios. Explainability is essential for gaining user trust in advanced optimization methods, particularly black box optimization. By providing clear, interpretable explanations of how an algorithm makes its decisions, stakeholders gain a better understanding of the process and are more likely to adopt and rely on these techniques. This transparency not only strengthens confidence in the outcomes but also makes it easier to identify potential biases or flaws. In SO, however, the complexity of both surrogate models and sampling strategies (for example, acquisition functions) often leads to a lack of clarity. While existing research has largely focused on improving convergence to global optima, the practical explainability of newly proposed strategies, particularly in batch evaluation settings, remains insufficiently explored. This thesis introduces novel approaches for balancing the exploration-exploitation trade-off by prioritizing diversity for batch sampling. These strategies prioritize diverse candidate batch generation through adaptive sampling techniques, infusing vitality into the optimization process and effectively exploring uncharted regions of the search space. Empirical validation demonstrates that these methods effectively navigate complex design landscapes, as shown by testing on diverse real-world benchmark problems, including DNA binding, Airfoil design, and MNIST hyperparameter optimization. Beyond theoretical advancements and empirical validation, this thesis lays the groundwork for a paradigm shift, empowering practitioners to approach complex optimization challenges with renewed precision by promoting diversity and elevated exploration. Additionally, the thesis explores the impact of SO's algorithmic hyperparameters on the exploration-exploitation trade-off to establish a robust framework. It aims to present a holistic view of surrogate optimization as a cohesive system, offering fresh insights into learning and optimizing hyperparameters across different frameworks without the need for manual tuning or incurring significant computational costs. Lastly, this thesis addresses the primary challenge of explainability in Surrogate Optimization by introducing a comprehensive, model-agnostic framework of explainability metrics that can potentially enhance user trust. These metrics provide intermediate and post-hoc explanations to guide practitioners in understanding the SO process before and after costly evaluations. To evaluate the impact of these metrics in real-world scenarios, we applied them to benchmarks such as Robot Pushing and rover trajectory challenges, observing how deeper insights can enhance user confidence in the optimization process.