Proactive human avoidance in collaborative logistics: an RL-based dynamic scheduling framework

The transition toward Human-Robot Collaboration (HRC) in modern manufacturing seeks to combine the cognitive flexibility of human operators with the precision and repeatability of robotic systems. However, introducing collaborative robots into shared workspaces introduces significant safety and efficiency challenges. Standard reactive safety protocols, such as Speed and Separation Monitoring (SSM), ensure safety by dynamically reducing the robot's speed or forcing a complete protective stop when a human approaches, frequently causing prolonged idle times and severe degradation of system throughput. This thesis addresses these inefficiencies within the specific domain of collaborative logistics and palletizing, where task allocation is often predetermined and the primary challenge is the optimal real time temporal sequencing and spatial coordination of operations. To resolve the inherent conflict between operational throughput and human safety, this work proposes an intelligent, dynamic task scheduling framework based on Deep Reinforcement Learning. The collaborative workstation is formally modeled as a Markov Decision Process (MDP), treating the human operator, governed by a probabilistic Finite State Machine, as an autonomous agent. A robotic orchestrator is trained using the Masked Proximal Policy Optimization (PPO) algorithm, integrated via a custom communication bridge with a high-fidelity Siemens Tecnomatix digital twin. Furthermore, dynamic action masking is implemented at every decision step to strictly enforce complex physical capacities and logical production constraints, effectively reducing the exploration space and accelerating algorithmic convergence. The simulation results demonstrate that the learned policy successfully achieves the dual-objective optimization: maximizing system efficiency while proactively minimizing spatial interference with the human operator. Instead of relying on reactive safety stops, the Reinforcement Learning agent exhibits proactive behaviors. By intelligently utilizing a deliberate Wait action to yield to the human, and employing Opposite Side Tasking to operate in zones away from the human's trajectory, the robot dynamically adapts its schedule. This proactive spatial deconfliction incentivizes parallel workflows and significantly reduces safety triggered stops, ultimately minimizing overall cycle time compared to traditional reactive setups. Overall, this thesis proves that RL-based orchestrators can effectively manage the strict spatial and temporal demands of collaborative logistics. Having validated these proactive avoidance strategies within a high-fidelity digital twin, these findings establish a pathway for future research focused on deploying the system to physical robotic workstations.

La Collaborazione Uomo-Robot (HRC) unisce la flessibilità cognitiva umana alla precisione robotica. Tuttavia, protocolli di sicurezza reattivi come lo Speed and Separation Monitoring (SSM) degradano frequentemente la produttività, forzando arresti o decelerazioni all'avvicinarsi dell'operatore. Per mitigare tali inefficienze, questa tesi propone un framework di schedulazione dinamica basato sul Deep Reinforcement Learning (DRL) per ottimizzare il coordinamento spaziale e temporale in tempo reale. La postazione condivisa è formalizzata come Processo Decisionale di Markov (MDP), dove l'umano è modellato come agente autonomo governato da una Macchina a Stati Finiti probabilistica. Un orchestratore robotico è addestrato tramite l'algoritmo Masked Proximal Policy Optimization all'interno di un digital twin (Siemens Tecnomatix Process Simulate). Un meccanismo di action-masking dinamico impone rigidi vincoli fisici e produttivi, riducendo drasticamente lo spazio di esplorazione e accelerando la convergenza. La policy appresa ottimizza un duplice obiettivo: minimizzare il tempo di ciclo e ridurre proattivamente le interferenze con l'operatore. Rispetto a una baseline reattiva basata sulla velocità, l'approccio proposto abbatte gli arresti di sicurezza indotti dall'SSM. Pur ottenendo un tempo di ciclo nominalmente inferiore, la strategia reattiva subisce continue interruzioni. Al contrario, l'orchestratore RL adotta strategie proattive, tra cui l'attesa strategica (Wait) e l'Opposite Side Tasking, garantendo operazioni parallele e una separazione stabile delle zone. Le simulazioni dimostrano che il framework preserva un'elevata produttività, eliminando virtualmente gli arresti di sicurezza durante le attività. Anticipando i movimenti umani anziché reagire alle violazioni di prossimità, l'orchestratore assicura un'esecuzione più fluida e un'interazione prevedibile, supportando una collaborazione efficiente.