In the last decades, Knowledge Graph (KG) empowered analytics have been used to extract advanced insights from data. Several companies have integrated their legacy relational databases with semantic technologies using Ontology-Based Data Access (OBDA), which enables analysts to write SPARQL queries both over KGs and SQL relational data sources by making transparent most of the implementation details. The exponential increase of the data volumes has led to a massive adoption of new large-scale analytical engines to store and process big volumes of relational data, such as Apache Spark. Moreover, Stream Processing (SP) engines such as Apache Flink enables to process big volumes of data in motion. However, there is a gap between big data and semantic technologies, as state of the art OBDA systems like Ontop are still focused on supporting only relational databases. The scope of this thesis is to extend the OBDA paradigm to big data and streaming scenarios. The Chimera suite development is motivated by both the needs of Ricerca sul Sistema Energetico S.p.A. and the goal to provide a set of open-source tools addressable to any scenario. OntopSpark is a big data OBDA extension of Ontop for performing KG-empowered analytics over data lakes managed by the Apache Spark query engine, while PySPARQL is a python library for materializing SPARQL responses as Spark DataFrames or GraphFrames in notebooks. Chimera enables round-tripping analyses, a new type of data science pipeline where users can query Spark data lakes with OntopSpark and PySPARQL, and save back semantically enriched results in Spark tables. OntopStream is an Ontop extension for performing Streaming-OBDA of relational data streams stored and managed by Apache Flink. The Streaming Virtual Knowledge Graph approach enables the definition of queryable RDF representations of relational streams without allocating additional space. RSP-QL queries can be used to perform KG-empowered continuous analyses based on the relational data streams stored in Flink dynamic tables. Moreover, the integration with Jupyter lets users developing automated analysis scripts in their notebooks.
Negli ultimi decenni, le analisi potenziate da Knowledge Graph (KG) sono state utilizzate per estrarre informazioni avanzate dai dati. Diverse aziende hanno integrato i propri database relazionali legacy con tecnologie semantiche per l'accesso ontologico alle basi di dati (OBDA), che consentono agli analisti di interrogare con query SPARQL sia i KG che le sorgenti dati relazionali SQL, rendendo trasparente la maggior parte dei dettagli implementativi. L'aumento esponenziale del volume dei dati ha portato a una massiccia adozione di nuovi engine analitici su larga scala per archiviare e gestire grandi volumi di dati relazionali, come Apache Spark. Inoltre, gli Stream Processing (SP) engines come Apache Flink permettono di processare grandi volumi di dati in movimento. Tuttavia, esiste un divario tra le tecnologie big data e semantiche, poiché anche i sistemi OBDA allo stato dell'arte come Ontop sono ancora focalizzati solamente al supporto di database relazionali. Lo scopo di questa tesi é di estendere il paradigma OBDA a scenari big data e streaming. Lo sviluppo della suite Chimera é motivato sia dalle esigenze di Ricerca sul Sistema Energetico S.p.A., sia dall'obiettivo di fornire un set di strumenti open-source utlizzabili in qualsiasi scenario. OntopSpark é un'estensione OBDA big data di Ontop per eseguire analisi potenziate da KG su data lake gestiti dall'engine Apache Spark, mentre PySPARQL é una libreria python per materializzare le risposte SPARQL sottoforma di Spark DataFrame o GraphFrame nei notebook. Chimera consente delle analisi circolari, una nuova tipologia di data pipeline per interrogare i data lake Spark con OntopSpark e PySPARQL, e salvare i risultati semanticamente arricchiti in tabelle Spark. OntopStream é un'estensione di Ontop per effettuare Streaming-OBDA su flussi di dati relazionali archiviati e gestiti da Apache Flink. L' approccio Streaming Virtual Knowledge Graph permette la definizione di rappresentazioni RDF interrogabili di flussi relazionali senza allocare ulteriore spazio. Le query RSP-QL possono essere usate per effettuare analisi continue potenziate da KG, basate su flussi di dati relazionali memorizzati in tabelle dinamiche di Flink. Inoltre, l'integrazione con Jupyter permette agli utenti di sviluppare script di analisi automatici nei notebook.
Streaming virtual knowledge graphs for real-time big data analytics
Belcao, Matteo
2020/2021
Abstract
In the last decades, Knowledge Graph (KG) empowered analytics have been used to extract advanced insights from data. Several companies have integrated their legacy relational databases with semantic technologies using Ontology-Based Data Access (OBDA), which enables analysts to write SPARQL queries both over KGs and SQL relational data sources by making transparent most of the implementation details. The exponential increase of the data volumes has led to a massive adoption of new large-scale analytical engines to store and process big volumes of relational data, such as Apache Spark. Moreover, Stream Processing (SP) engines such as Apache Flink enables to process big volumes of data in motion. However, there is a gap between big data and semantic technologies, as state of the art OBDA systems like Ontop are still focused on supporting only relational databases. The scope of this thesis is to extend the OBDA paradigm to big data and streaming scenarios. The Chimera suite development is motivated by both the needs of Ricerca sul Sistema Energetico S.p.A. and the goal to provide a set of open-source tools addressable to any scenario. OntopSpark is a big data OBDA extension of Ontop for performing KG-empowered analytics over data lakes managed by the Apache Spark query engine, while PySPARQL is a python library for materializing SPARQL responses as Spark DataFrames or GraphFrames in notebooks. Chimera enables round-tripping analyses, a new type of data science pipeline where users can query Spark data lakes with OntopSpark and PySPARQL, and save back semantically enriched results in Spark tables. OntopStream is an Ontop extension for performing Streaming-OBDA of relational data streams stored and managed by Apache Flink. The Streaming Virtual Knowledge Graph approach enables the definition of queryable RDF representations of relational streams without allocating additional space. RSP-QL queries can be used to perform KG-empowered continuous analyses based on the relational data streams stored in Flink dynamic tables. Moreover, the integration with Jupyter lets users developing automated analysis scripts in their notebooks.| File | Dimensione | Formato | |
|---|---|---|---|
|
2021_10_Belcao.pdf
accessibile in internet per tutti
Descrizione: thesis
Dimensione
4.11 MB
Formato
Adobe PDF
|
4.11 MB | Adobe PDF | Visualizza/Apri |
I documenti in POLITesi sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/10589/179356