today, there are many data sources—such as IoT devices, user interaction events from mobile applications, financial service transactions, and health monitoring systems—that broadcast critical information in real time. Developers working with these data sources need to think about the architecture to capture real-time streaming data at varying scales and complexities.

It used to be that processing real-time information at significant scale was hard to implement. Hardware architectures needed to be engineered for low latency while software needed more advanced programming techniques that combined receiving data, processing it, and shipping it efficiently.