Applying Reinforcement Learning (RL) in complex domains such as internal industrial logistics and energy grid management is hindered by the prohibitive computational cost of high-fidelity simulations. This bottleneck slows down experimentation and renders advanced algorithms that hybridize planning and learning, such as AlphaZero, infeasible in realistic scenarios. This thesis confronts this challenge by developing a Model-Based Reinforcement Learning framework. The research arc begins in a known-model paradigm to establish a baseline, and progresses to the main contribution: learning unknown and physically consistent environment dynamics through surrogate models. Initially, Storehouse, a standardized and scalable simulation environment for warehouse management, is developed. Scaling this environment to a realistic complexity revealed the computational bottlenecks that motivated the shift towards surrogate modeling. To overcome the limitations of traditional sampling in high-dimensional state spaces, the thesis introduces an innovative agent-based trajectory sampling methodology. It is demonstrated that a mixed sampling strategy, leveraging expert, random, and maximum-entropy agents, is essential for generating robust and representative datasets. However, purely empirical surrogates often fail to guarantee physical consistency, leading to compounding errors. To solve this, the core of this work proposes and validates the use of Physics-Informed Neural Networks (PINNs). By integrating the governing differential equations of the system directly into the network's loss function, PINNs generate models that are both physically consistent and highly generalizable. The complete framework is validated in a complex smart grid management environment. The PINN-based surrogate proved to be the only model capable of training a stable and effective control policy, reducing training time by up to 50% while maintaining performance comparable to the original simulator. Our analysis shows that PINNs decisively outperform state-of-the-art models like XGBoost in generalization and long-term dynamic stability. This thesis contributes a validated and efficient method that bridges the gap between high-fidelity simulations and advanced learning algorithms, making previously intractable optimization problems feasible.
L'applicazione del Reinforcement Learning (RL) in domini complessi come la logistica industriale interna e la gestione delle reti energetiche è ostacolata dal costo computazionale proibitivo delle simulazioni ad alta fedeltà. Questo collo di bottiglia rallenta la sperimentazione e rende impraticabili in scenari realistici algoritmi avanzati che ibridano pianificazione e apprendimento, come AlphaZero. Questa tesi affronta tale sfida sviluppando un framework di Reinforcement Learning Model-Based. Il percorso di ricerca parte da un paradigma a modello noto per stabilire una baseline, e progredisce fino al contributo principale: l'apprendimento di dinamiche ambientali sconosciute e fisicamente consistenti attraverso modelli surrogati. Inizialmente viene sviluppato Storehouse, un ambiente di simulazione standardizzato e scalabile per la gestione del magazzino. Il ridimensionamento di questo ambiente verso una complessità realistica ha rivelato i colli di bottiglia computazionali che hanno motivato lo spostamento verso la modellazione surrogata. Per superare i limiti del campionamento tradizionale in spazi di stato ad alta dimensionalità, la tesi introduce una metodologia innovativa di campionamento di traiettorie basata su agenti. Si dimostra che una strategia di campionamento mista, che sfrutta agenti esperti, casuali e a massima entropia, è essenziale per generare dataset robusti e rappresentativi. Tuttavia, i surrogati puramente empirici spesso non riescono a garantire la consistenza fisica, portando ad errori cumulativi. Per risolvere questo problema, il nucleo del lavoro propone e valida l'uso delle Physics-Informed Neural Networks (PINN). Integrando le equazioni differenziali che governano il sistema direttamente nella funzione di loss della rete, le PINN generano modelli che sono al contempo fisicamente consistenti e altamente generalizzabili. Il framework completo viene validato in un ambiente complesso di gestione di una smart grid. Il surrogato basato su PINN si è rivelato l'unico modello in grado di addestrare una policy di controllo stabile ed efficace, riducendo il tempo di addestramento fino al 50% mantenendo prestazioni paragonabili al simulatore originale. La nostra analisi mostra che le PINN superano in modo decisivo modelli allo stato dell'arte come XGBoost in termini di generalizzazione e stabilità dinamica a lungo termine. Questa tesi contribuisce con un metodo validato ed efficiente che colma il divario tra simulazioni ad alta fedeltà e algoritmi di apprendimento avanzati, rendendo praticabili problemi di ottimizzazione precedentemente intrattabili.
Bridging the gap between Reinforcement Learning and planning
CESTERO PORTU, JULEN CARLOS
2025/2026
Abstract
Applying Reinforcement Learning (RL) in complex domains such as internal industrial logistics and energy grid management is hindered by the prohibitive computational cost of high-fidelity simulations. This bottleneck slows down experimentation and renders advanced algorithms that hybridize planning and learning, such as AlphaZero, infeasible in realistic scenarios. This thesis confronts this challenge by developing a Model-Based Reinforcement Learning framework. The research arc begins in a known-model paradigm to establish a baseline, and progresses to the main contribution: learning unknown and physically consistent environment dynamics through surrogate models. Initially, Storehouse, a standardized and scalable simulation environment for warehouse management, is developed. Scaling this environment to a realistic complexity revealed the computational bottlenecks that motivated the shift towards surrogate modeling. To overcome the limitations of traditional sampling in high-dimensional state spaces, the thesis introduces an innovative agent-based trajectory sampling methodology. It is demonstrated that a mixed sampling strategy, leveraging expert, random, and maximum-entropy agents, is essential for generating robust and representative datasets. However, purely empirical surrogates often fail to guarantee physical consistency, leading to compounding errors. To solve this, the core of this work proposes and validates the use of Physics-Informed Neural Networks (PINNs). By integrating the governing differential equations of the system directly into the network's loss function, PINNs generate models that are both physically consistent and highly generalizable. The complete framework is validated in a complex smart grid management environment. The PINN-based surrogate proved to be the only model capable of training a stable and effective control policy, reducing training time by up to 50% while maintaining performance comparable to the original simulator. Our analysis shows that PINNs decisively outperform state-of-the-art models like XGBoost in generalization and long-term dynamic stability. This thesis contributes a validated and efficient method that bridges the gap between high-fidelity simulations and advanced learning algorithms, making previously intractable optimization problems feasible.| File | Dimensione | Formato | |
|---|---|---|---|
|
PhD_Thesis-1.pdf
accessibile in internet per tutti
Descrizione: Applying Reinforcement Learning (RL) in complex domains such as internal industrial logistics and energy grid management is hindered by the prohibitive computational cost of high-fidelity simulations. This bottleneck slows down experimentation and renders advanced algorithms that hybridize planning and learning, such as AlphaZero, infeasible in realistic scenarios. This thesis confronts this challenge by developing a Model-Based Reinforcement Learning framework. The research arc begins in a known-model paradigm to establish a baseline, and progresses to the main contribution: learning unknown and physically consistent environment dynamics through surrogate models. Initially, Storehouse, a standardized and scalable simulation environment for warehouse management, is developed. Scaling this environment to a realistic complexity revealed the computational bottlenecks that motivated the shift towards surrogate modeling. To overcome the limitations of traditional sampling in high-dimensional state spaces, the thesis introduces an innovative agent-based trajectory sampling methodology. It is demonstrated that a mixed sampling strategy, leveraging expert, random, and maximum-entropy agents, is essential for generating robust and representative datasets. However, purely empirical surrogates often fail to guarantee physical consistency, leading to compounding errors. To solve this, the core of this work proposes and validates the use of Physics-Informed Neural Networks (PINNs). By integrating the governing differential equations of the system directly into the network's loss function, PINNs generate models that are both physically consistent and highly generalizable. The complete framework is validated in a complex smart grid management environment. The PINN-based surrogate proved to be the only model capable of training a stable and effective control policy, reducing training time by up to 50% while maintaining performance comparable to the original simulator. Our analysis shows that PINNs decisively outperform state-of-the-art models like XGBoost in generalization and long-term dynamic stability. This thesis contributes a validated and efficient method that bridges the gap between high-fidelity simulations and advanced learning algorithms, making previously intractable optimization problems feasible.
Dimensione
5.93 MB
Formato
Adobe PDF
|
5.93 MB | Adobe PDF | Visualizza/Apri |
I documenti in POLITesi sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/10589/256897