Data Streaming Cloud Applications

Building Data Platform

What is the purpose of streaming data?

1. Time-critical decisions
2. Business intelligence

Time value to the data will help businesses to take decisions on drive efficiency, save cost, etc.

Data streaming varies from →milliseconds → to seconds→to Minutes

Data streaming in milliseconds use cases:

1. Push notifications
2. Messages between services

Data streaming in seconds use cases:

1. Customer experience logs
2. Infrastructure logs
3. Application logs
4. Security logs
5. IoT device logs
6. CDS(Change data capture)

Data streaming in minutes use cases:

1. Data lakes
2. Data warehouse

Enabling real-time analytics:

Data streaming enables customers to ingest, process, and analyze high volumes of high-velocity data from a variety of sources in real-time.

AWS uses Kinesis for real-time streaming

Kinesis Data Streams: Collects and stores data stream for real-time analytics.

Kinesis Data Firehose: Loads data streams into AWS resources.

Kinesis Data Analytics: Analyze data stream with SQL or Java.

Stream Producers:

1. Mobile Apps
2. Web Clickstream
3. Application Logs
4. Metering records
5. IoT sensors
6. Smart Buildings

Stream Ingestion:

1. Toolkit and Libraries
2. Service Integrations

Stream Storage:

Data is stored in the order it was received for a set duration of time and can be replayed indefinitely at this time.

Kinesis Data Streams vs Data Firehose:

Kinesis data stream used for custom processing, per incoming record, with sub-second processing latency and allows the choice of stream processing frameworks.

Kinesis Data Firehose used for serverless capability and use existing analytical tools and with the latency of 60 seconds or more

Stream processing:

Records are read in the order they produced, enabling real-time analytics or streaming ETL.

Kinesis Data Analytics SQL use cases:

1. Sub-second end-to-end processing latencies
2. SQL steps can be chained together for parallel or serial steps
3. Build applications with several queries
4. Prebuild functions like sum and count distinct
5. Running aggregations continuously.

Kinesis Data Analytics JAVA use cases:

1. Sophisticated Applications
2. Uses Apache Flink engine for stateful processing of the data
3. Strong data integrity

Uses cases Data categorization:

1. Logs: Application or service logs
2. eCommerce: Data describes what customer has purchased
3. Interactions: How customers interact with the website
4. Products: Web site content

Challenges with Data Platform Design:

1. Service re-architecture
2. slow and unreliable data processing
3. Built for humans vs built as a service
4. Private data center limitations

Considerations:

1. Migrating to cloud
2. Treating data as a service
3. Low latency for data processing

Why Streaming?

1. Business evaluation requires richer and more timely data
2. Solving query across multiple microservices
3. Machine learning

Streaming workloads:

1. API for products: <5 seconds end-to-end latency, Enrichment of data, and unified data model
2. Customer notifications: <5 minutes end-to-end latency, PII and data compliance, and exactly-once delivery
3. Analytics: <5 minutes latency

Stateful transformation use case:

Stateful transformation is required for events which need to produce aggregated results based on event happened in past. Example counting number of orders by existing orders and new order event. These kinds of results are achieved by using stateful data transformation by storing existing events in memory. Appending aggregated result to the event and storing it in a DB will enable accessing results using API.

Goals:

1. Single data platform for marketing and APIs for products
2. Low-latency, stateful streaming
3. <15 seconds end-to-end data latency across all scenarios
4. Engineering excellence using CI/CD, unit, integration and functional testing, DR and rebuild capabilities, monitoring, and alerting.

Replay events data platform use cases:

1. If business logic changes replay and execute all of the events from day zero
2. Disaster recovery if we lost the database restore the database by re-running events from the data platform with all of the events from day zero
3. If we need to add new integration like Redshift we need to flow all data history along with the latest data