Google·hard

Design a Distributed Metrics Logging and Aggregation System

Design a distributed system to collect, store, and aggregate metrics from thousands of services in real-time, challenging due to high ingestion rates, time-series storage, and query performance at petabyte scale.

Functional

▸Collect metrics data from multiple sources such as App Server and Mobile App
▸Store and process large volumes of metrics data in real-time
▸Provide querying and filtering capabilities for metrics data
▸Support multiple data formats such as JSON and CSV
▸Handle errors and exceptions in metrics data collection and processing

Non-Functional

▸Handle 100,000 requests per second with average latency of 50ms
▸Achieve 99.99% availability with less than 5 minutes of downtime per year
▸Store at least 1 year of metrics data with total storage capacity of 100TB
▸Support 10,000 concurrent users querying metrics data
▸Process 1 million metrics data points per second with less than 1% data loss

Add at least 2 components to run