Amazon·hard

Design a Distributed Stream Processing System like Kafka

Design a distributed stream processing system like Kafka to handle high-volume and high-velocity data streams, challenging due to the need for fault tolerance, exactly-once delivery guarantees, and horizontal scalability.

Functional

▸Handle 10,000 messages per second from multiple sources
▸Support multiple data formats such as JSON, Avro, and Protobuf
▸Provide exactly-once processing guarantees for messages
▸Allow for real-time data aggregation and filtering
▸Support integration with multiple downstream systems for data consumption

Non-Functional

▸Achieve end-to-end latency of less than 100ms for 99.9% of messages
▸Ensure system availability of at least 99.99% with less than 5 minutes of downtime per year
▸Handle peak throughput of 100,000 messages per second with less than 1% message loss
▸Support 10,000 concurrent connections from producers and consumers
▸Guarantee data durability with less than 1% data loss in the event of a failure

Add at least 2 components to run