EL-3ashmawy
7.8K posts

EL-3ashmawy
@AcountUnknown9
هذا الحساب برئ من السيسي و زبانيته امام ما يحدث في غزه و امام كل ظلم وقع على اي مظلوم


🧵 Day 20/30 — #SystemDesign Kafka: The backbone behind real-time data pipelines at scale Most systems don’t just serve requests. They produce continuous streams of events — user clicks, payments, logs, metrics, notifications. Handling this data reliably and at scale is not trivial. That’s where Apache Kafka comes in. Kafka is a distributed event streaming platform used to build real-time data pipelines and streaming applications. It allows services to publish and consume events efficiently without tight coupling. ---------------------- Core Idea !! Instead of services calling each other directly: → Producer sends event to Kafka → Kafka stores it durably → Multiple consumers read and process independently Flow: Producer → Kafka Topic → Consumers One event can power many systems at once. --------------------- Key Concepts 1. Topic A category of events (e.g., orders, payments, logs) 2. Partition Topics are split into partitions for parallel processing 3. Producer Sends messages to Kafka 4. Consumer Reads messages from Kafka 5. Consumer Group Multiple consumers sharing load 6. Offset Position of message in partition These concepts define Kafka’s power. ------------------------ Why Kafka Matters → High throughput (millions of messages/sec) → Fault tolerant (replication across brokers) → Scalable horizontally → Durable storage (events persisted) → Real-time processing → Decouples systems It turns data into a streaming backbone. --------------------------- Real-World Example E-commerce order placed: → Event sent to Kafka (OrderCreated) Consumers: → Payment service processes transaction → Inventory updates stock → Notification sends email → Analytics tracks event → Recommendation system updates behavior All from one event stream. -------------------------------- Why Companies Use Kafka → Netflix for event streaming pipelines → LinkedIn (creator of Kafka) → Uber for real-time data flows → Amazon for internal streaming systems → Fintech apps for transaction streams Kafka powers modern data-driven systems. -------------------------------- Important Strength : Kafka stores events, not just forwards them. Consumers can: → Read in real-time → Replay old events → Recover from failures → Process at their own pace This makes systems resilient. ---------------------------- Challenges Most Ignore Kafka is powerful, but not simple: → Requires cluster management → Partition design is critical → Ordering only within partition → Exactly-once semantics is complex → Monitoring and tuning needed Misuse leads to complexity quickly. --------------- Kafka vs Queue Queue: → Message consumed once Kafka: → Message stored + can be consumed multiple times Kafka is more like a log system than a simple queue. Don’t use it for simple request-response systems. #30DaysOfSystemDesign #Kafka #BackendEngineering





Kubernetes Simplified: Most people get overwhelmed by K8s. Here’s the simplest way to understand it 👇 Part1 (1-5) @techfusionist/note/c-247306631?r=4cjx17&utm_medium=ios&utm_source=notes-share-action" target="_blank" rel="nofollow noopener">substack.com/@techfusionist…
Part1 (6-10) @techfusionist/note/c-251142337?r=4cjx17&utm_medium=ios&utm_source=notes-share-action" target="_blank" rel="nofollow noopener">substack.com/@techfusionist…











Distributed Systems I will share the video insights.












