The need to build Machine Learning (ML) models based on large amounts of data brought new challenges to data management systems. Feature stores have emerged as a centralized data platform enabling feature reuse while ensuring consistency between feature engineering, model training, and inference. Recent publications demonstrate that the Hopsworks feature store outperforms existing cloud-based alternatives in training and online inference query workloads. In its offline feature store, the Hopsworks feature store stores batch or historical data, collecting it into feature groups organized in Apache Hudi tables and stored on HopsFS, Hopsworks HDFS distribution. However, even in this system, the latency to perform a write operation is at least one or more minutes, even for small quantities of data (1 GB or less). The hypothesis of this work is that the limitation is caused by Spark, which the system uses to write data on Apache Hudi tables. A promising approach to avoid using Spark appears to be adopting Delta Lake instead of Apache Hudi and access data using a Rust library called delta-rs. This thesis investigates the possibility of reducing the read and write latency in the offline feature store by expanding the delta-rs library to support HDFS and HopsFS and comparatively evaluating the performance of the legacy and newly implemented system. Two major iterations of storage support in delta-rs for HopsFS were developed to meet the strict production-ready requirements. The system was then evaluated by performing and measuring read and write operations increasing the number of CPU cores up to eight. Results confirmed the superior performance of the delta-rs library over the Spark system in all write operations with a latency reduction from ten up to forty times. Delta-rs also surpassed the Spark alternative in read operations with a latency reduction of forty-seven percent, up to forty times. These findings encourage future research investigating Spark alternative when optimizing perfor- mance in small-scale (1 GB - 100 GB) data management systems. The system developed will find application in the Hopsworks feature store production environment.
La necessità di costruire modelli di Machine Learning (ML) basati su grandi quantità di dati ha portato nuove sfide ai sistemi di gestione dei dati. I feature stores sono emersi come piattaforma dati centralizzata che consente il riutilizzo delle features. Recenti pubblicazioni dimostrano che il feature store di Hopsworks supera le alternative cloud esistenti. Nel suo offline feature store, Hopsworks memorizza dati batch o storici, raccogliendoli in feature groups organizzati in tabelle Apache Hudi e archiviati su HopsFS, la distribuzione HDFS di Hopsworks. Tuttavia, anche in questo sistema, la latenza per eseguire un’operazione di scrittura è di almeno uno o più minuti, anche per piccole quantità di dati (1 GB o meno). L’ipotesi di questa tesi è che la limitazione sia causata da Spark, che il sistema utilizza per scrivere i dati sulle tabelle di Apache Hudi. Un approccio promettente per evitare l’uso di Spark sembra essere l’adozione di Delta Lake al posto di Apache Hudi e l’accesso ai dati utilizzando una libreria Rust chiamata delta-rs. Questa tesi studia la possibilità di ridurre la latenza di lettura e scrittura nel feature store offline espandendo la libreria delta-rs per supportare HDFS e HopsFS e valutando in modo comparativo le prestazioni del sistema preesistente e di quello appena imple- mentato. Sono state sviluppate due distinte implementazioni per aggiungere supporto in delta-rs per HopsFS, a causa dei severi requisiti di produzione. Il sistema è stato poi valutato eseguendo e misurando le operazioni di lettura e scrittura. I risultati hanno confermato la superiorità delle prestazioni della libreria delta-rs rispetto al sistema Spark in tutte le operazioni di scrittura, con una riduzione della latenza da dieci a quaranta volte. Delta-rs ha superato l’alternativa Spark anche nelle operazioni di lettura, con una riduzione della latenza del 47%, fino a quaranta volte. Questi risultati incoraggiano la ricerca futura sull’alternative a Spark per l’ottimizzazione delle prestazioni nei sistemi di gestione dei dati di piccola dimensione (1 GB - 100 GB). Il sistema sviluppato troverà applicazione nell’ambiente di produzione del feature store di Hopsworks.
Reducing read and write latency in a Delta Lake-backed offline feature store
MANFREDI, GIOVANNI
2023/2024
Abstract
The need to build Machine Learning (ML) models based on large amounts of data brought new challenges to data management systems. Feature stores have emerged as a centralized data platform enabling feature reuse while ensuring consistency between feature engineering, model training, and inference. Recent publications demonstrate that the Hopsworks feature store outperforms existing cloud-based alternatives in training and online inference query workloads. In its offline feature store, the Hopsworks feature store stores batch or historical data, collecting it into feature groups organized in Apache Hudi tables and stored on HopsFS, Hopsworks HDFS distribution. However, even in this system, the latency to perform a write operation is at least one or more minutes, even for small quantities of data (1 GB or less). The hypothesis of this work is that the limitation is caused by Spark, which the system uses to write data on Apache Hudi tables. A promising approach to avoid using Spark appears to be adopting Delta Lake instead of Apache Hudi and access data using a Rust library called delta-rs. This thesis investigates the possibility of reducing the read and write latency in the offline feature store by expanding the delta-rs library to support HDFS and HopsFS and comparatively evaluating the performance of the legacy and newly implemented system. Two major iterations of storage support in delta-rs for HopsFS were developed to meet the strict production-ready requirements. The system was then evaluated by performing and measuring read and write operations increasing the number of CPU cores up to eight. Results confirmed the superior performance of the delta-rs library over the Spark system in all write operations with a latency reduction from ten up to forty times. Delta-rs also surpassed the Spark alternative in read operations with a latency reduction of forty-seven percent, up to forty times. These findings encourage future research investigating Spark alternative when optimizing perfor- mance in small-scale (1 GB - 100 GB) data management systems. The system developed will find application in the Hopsworks feature store production environment.| File | Dimensione | Formato | |
|---|---|---|---|
|
reducing_read_and_write_latency_master_thesis_polimi.pdf
accessibile in internet per tutti
Descrizione: Thesis file
Dimensione
4.43 MB
Formato
Adobe PDF
|
4.43 MB | Adobe PDF | Visualizza/Apri |
I documenti in POLITesi sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/10589/229575