Nowadays streaming is part of our daily lives, one of the most relevant open-source frameworks for stream processing is Apache Flink, and considering the most known properties and limitations of Big Data (velocity, variety, veracity, volume), it is understandable to wonder whether the research fields like semantic web technologies may help with Data Integration (variety) limitations in a streaming world. By using OBDA (Ontology Based Data Access) engines it is possible to tackle some of those limitations, however, only a few currently support streaming data sources. Thanks to the release of the SQL gateway and JDBC driver for Apache Flink, the first stage of the integration for Flink Dynamic Tables and an OBDA system became possible. OntopStream is one of the technologies that benefited from it, Ontop is an Ontology Base Data Access engine that allows performing SPARQL queries over relational databases, and has recently been integrated to support Big Data Technologies like Apache Spark, Databricks, Snowflake, and more. OntopStream is an extension of Ontop that is introducing Apache Flink into the supported systems. The objective of this thesis is to illustrate the integration of OntopStream with joins over Apache Flink Dynamic Tables. Dynamic Tables are the core of Apache Flink and can be queried with SQL-like queries, yielding continuous queries over stream messages, hence, producing dynamic results. Differently from the classical databases, in the streaming world, tables are not fixed a priori from the execution of the query, therefore it is critical to be aware of the operational implications of a Regular Join over unbounded tables. This thesis will represent the alternatives to regular joins and the feasibility to integrate Interval Joins into OntopStream. Finally, an emerging streaming python library that was used to perform RSP-QL operations from Python, and its integration into the standard Package-Management System recommended by the Python Software Foundation, will be presented at the end of the thesis.
Oggigiorno lo streaming fa parte delle nostre vite quotidiane, uno dei framework più rilevanti per lo stream processing è Apache Flink, e considerando le proprietà più note del mondo Big Data (velocità, varietà, veracità, volume), è facile chiedersi se le tecnologie semantic web potrebbero aiutare nei limiti di data integration (varietà) in un mondo di streaming. Utilizzando sistemi OBDA (Ontology Based Data Access) è possibile affrontare questi limiti, anche se al momento solo pochi supportano sorgenti in streaming. Grazie al rilascio dello SQL gateway e il relativo JDBC driver per Apache Flink, è stato possible inizare una prima fase di integrazione tra le Flink Dynamtic Tables e gli Ontology system. Una delle tecnologie che ha beneficiato del loro rilascio è OntopStream. Ontop è un sistema OBDA che permette di effettuare query SPARQL su tabelle relazionali ed è stato recentemente integrato per supportare alcune delle più note tecnologie di Big Data come Apache Spark, Databricks, Snowflake, etc. OntopStream è un'estensione di Ontop che sta introducento Apache Flink tra i sistemi supportati. L'obiettivo di questa tesi è di illustrare l'integrazione tra Ontop e le join su Apache Flink Dynamic Tables. Le Dynamic Tables sono il cuore di Apache Flink e possono essere interrogate con queries SQL-like, portando alle continuous query su stream messages, di conseguenza, producendo risultati dinamici. Diversamente dai classici database, nel mondo dello streaming, le tabelle non sono fisse prima dell'esecuzione della query, per cui diventa critico essere consapevoli delle implicazioni operazionali che un Regular Join ha sulle tabelle non statiche e di illimitata dimensione. Si presenterà inoltre, i diversi join che esistono su Apache Flink e la fattibilità dell'integrazione degli Interval Join con OntopStream. In fine, l'emergente libreria python che è stata utilizzata per effettuare operazioni RSP-QL da python, e la sua integrazione sul sistema standard di Package-Management racommandato dalla Python Software Foundation sarà presentata alla fine della tesi.
Analyses of multi-windowing extension approaches for the Streaming Virtual Knowledge Graph tool Ontop-Stream
SALAZAR MOLINA, STEVEN ALEXANDER
2021/2022
Abstract
Nowadays streaming is part of our daily lives, one of the most relevant open-source frameworks for stream processing is Apache Flink, and considering the most known properties and limitations of Big Data (velocity, variety, veracity, volume), it is understandable to wonder whether the research fields like semantic web technologies may help with Data Integration (variety) limitations in a streaming world. By using OBDA (Ontology Based Data Access) engines it is possible to tackle some of those limitations, however, only a few currently support streaming data sources. Thanks to the release of the SQL gateway and JDBC driver for Apache Flink, the first stage of the integration for Flink Dynamic Tables and an OBDA system became possible. OntopStream is one of the technologies that benefited from it, Ontop is an Ontology Base Data Access engine that allows performing SPARQL queries over relational databases, and has recently been integrated to support Big Data Technologies like Apache Spark, Databricks, Snowflake, and more. OntopStream is an extension of Ontop that is introducing Apache Flink into the supported systems. The objective of this thesis is to illustrate the integration of OntopStream with joins over Apache Flink Dynamic Tables. Dynamic Tables are the core of Apache Flink and can be queried with SQL-like queries, yielding continuous queries over stream messages, hence, producing dynamic results. Differently from the classical databases, in the streaming world, tables are not fixed a priori from the execution of the query, therefore it is critical to be aware of the operational implications of a Regular Join over unbounded tables. This thesis will represent the alternatives to regular joins and the feasibility to integrate Interval Joins into OntopStream. Finally, an emerging streaming python library that was used to perform RSP-QL operations from Python, and its integration into the standard Package-Management System recommended by the Python Software Foundation, will be presented at the end of the thesis.| File | Dimensione | Formato | |
|---|---|---|---|
|
Thesis.pdf
accessibile in internet solo dagli utenti autorizzati
Descrizione: Analyses of multi-windowing extension approaches for the Streaming Virtual Knowledge Graph tool Ontop-Stream
Dimensione
1.47 MB
Formato
Adobe PDF
|
1.47 MB | Adobe PDF | Visualizza/Apri |
I documenti in POLITesi sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
https://hdl.handle.net/10589/203098