Active Fix Lite: A Time-Efficient Auto Encoder Based Non-FIFO Sliding Window Aggregation Approach for Real-Time Analytics

Abstract

In the fast dynamic digital world, the progress of stream processing technology experiences a dramatic shift in business operations transforming from a delayed periodic analysis to a continuous real-time insight on the fly. The ability to process this “data inmotion “has had a significant impact on all the real-time applications. Stream processing marks its foot print in almost all the application use cases like real-time fraud detection, prediction maintenance, enhanced customer experience, dynamic pricing and so on.Instream processing, First-In First-Out (FIFO) and non-FIFO streams are the two basic paradigms for the data to be processed challengingly based on the arrival order. Statistical stream aggregation is a crucial process in business analytics that summarizes and calculates high volume data using Sliding Window Aggregation (SWAG) techniques. In stream processing, there is a frequent possibility of occurrence of non-FIFO streams due to network issues. This research work introduces a time-efficient non-FIFO SWAG, Active Fix Lite which is an invariant of Active Fix technique that handles non-FIFO streams for a statistical stream aggregation process utilizing auto encoders as a dimensionality reduction tool to reduce the computational load that will be beneficial for streaming applications.

KEYWORDS

Sliding window aggregation, Non-FIFO streams, Auto encoder, Dimensionality reduction, Check points