Big Data for Real-Time Processing on Streaming Data: State-of-the-art and Future Challenges

Abstract

As many commercial businesses aspire for a competitive edge, real-time analysis on streaming data employing a big data methodology has lately become widespread. The ability to efficiently arrange massive amounts of data to make a business decision empowers data warehousing. Dealing with this sort of data poses considerable obstacles; as a result, several ways of assessing data in the form of streams have been created. Many solutions for dealing with enormous volumes of data and making decisions based on off-line batch processing have been investigated. In this paper, we explore the most recent developments in big data approaches for real-time analysis on streaming data to answer four research questions related to streaming data source, stream pre-processing, types of streaming data processing, used machine learning model, in addition to the validation and evaluation criteria. The system architecture associated with the used big data tools, the architecture hardware specs as well as other current platforms appropriate for large data streaming analytics are also considered. Furthermore, we outline numerous difficulties encountered in big data stream processing. The purpose of this review is to fill a gap in the surveyed area by offering thorough evaluation of various big data frameworks, architectures, and categories for cutting-edge approaches, as well as critical analyses of their performance and discussions of their applications, trends, and future directions to serve as guides for readers in this burgeoning study area.