TY - JOUR

T1 - A computational framework based on the dynamic pipeline approach

AU - Pasarella, Edelmira

AU - Vidal, Maria Esther

AU - Zoltan, Cristina

AU - Royo Sales, Juan Pablo

N1 - Funding Information: Edelmira Pasarella is partially supported by MCIN/AEI/10.13039/501100011033 under grant PID2020-112581GB-C21. Maria-Esther Vidal is partially supported by Leibniz Best Minds: Programme for Women Professors project TrustKG-Transforming Data in Trustable Insights with grant P99/2020.

PY - 2024/4/22

Y1 - 2024/4/22

N2 - Stream processing has inspired new computational approaches to facilitate effectiveness and efficiency. One such approach is the dynamic pipeline, which serves as a powerful computational model for stream processing. It is particularly well suited for solving problems that require incremental generation of results, making it an approach for scenarios where real-time analysis and responsiveness are critical. This paper aims to address a family of problems using the Dynamic Pipeline approach, and as a first step, we provide a comprehensive characterization of this problem family. In addition, we present the definition of a Dynamic Pipeline framework. To demonstrate the practicality of this framework, we present a proof of concept through its implementation and perform an empirical performance study. To this end, we focus on solving the problem of enumerating or listing the weakly connected components of a graph within the proposed framework. We provide two implementations of this algorithm to demonstrate the computational power and continuous behavior of the Dynamic Pipeline framework. The first implementation serves as a baseline for our experiments, representing an ad hoc solution based on the Dynamic Pipeline approach. In contrast, the second implementation is built on top of the developed framework. The observed results strongly support the suitability and effectiveness of the Dynamic Pipeline framework for implementing graph stream processing problems, especially those where continuous and real-time result generation is essential.

AB - Stream processing has inspired new computational approaches to facilitate effectiveness and efficiency. One such approach is the dynamic pipeline, which serves as a powerful computational model for stream processing. It is particularly well suited for solving problems that require incremental generation of results, making it an approach for scenarios where real-time analysis and responsiveness are critical. This paper aims to address a family of problems using the Dynamic Pipeline approach, and as a first step, we provide a comprehensive characterization of this problem family. In addition, we present the definition of a Dynamic Pipeline framework. To demonstrate the practicality of this framework, we present a proof of concept through its implementation and perform an empirical performance study. To this end, we focus on solving the problem of enumerating or listing the weakly connected components of a graph within the proposed framework. We provide two implementations of this algorithm to demonstrate the computational power and continuous behavior of the Dynamic Pipeline framework. The first implementation serves as a baseline for our experiments, representing an ad hoc solution based on the Dynamic Pipeline approach. In contrast, the second implementation is built on top of the developed framework. The observed results strongly support the suitability and effectiveness of the Dynamic Pipeline framework for implementing graph stream processing problems, especially those where continuous and real-time result generation is essential.

KW - Dynamic pipeline

KW - Parallelism

KW - Stream processing frameworks

UR - http://www.scopus.com/inward/record.url?scp=85190994512&partnerID=8YFLogxK

U2 - 10.1016/j.jlamp.2024.100966

DO - 10.1016/j.jlamp.2024.100966

M3 - Article

AN - SCOPUS:85190994512

VL - 139

JO - Journal of Logical and Algebraic Methods in Programming

JF - Journal of Logical and Algebraic Methods in Programming

SN - 2352-2208

M1 - 100966

ER -