Abstract message queue illustration

Kafka Reliability Playbook

Producers acks=all, min.insync.replicas>=2, idempotent producer on; enable transactions for exactly-once pipelines. Tune batch.size/linger.ms for throughput; cap max.in.flight.requests.per.connection for ordering. Handle retries with backoff; surface delivery errors. Consumers enable.auto.commit=false; commit after processing; DLT for poison messages. Size max.poll.interval.ms to work time; bound max.poll.records. Isolate heavy work in worker pool; keep poll loop fast. Topics & brokers RF ≥ 3; clean-up policy fit (delete vs compact); segment/retention sized to storage. Monitor ISR, under-replicated partitions, controller changes, request latency, disk usage. Throttle large produce/fetch; use quotas per client if needed. Checklist Producers idempotent, acks=all, MISR set. Consumers manual commit + DLT. RF/retention sized; ISR/URP monitored. Alerts on broker latency/disk/replication health.

March 5, 2025 · 4162 views
Abstract message queue illustration

RabbitMQ High Availability & Tuning

Queue types Prefer quorum queues for HA; classic for transient/high-throughput if loss acceptable. Set durability/persistence appropriately; avoid auto-delete for critical flows. Flow control Enable publisher confirms; set mandatory flag to catch unroutable messages. Use basic.qos to bound unacked messages; prefetch tuned per consumer. Watch memory/flow events; avoid oversized messages—use blob storage for big payloads. Topology & ops Mirror/quorum across AZs; avoid single-node SPOF. Use consistent hash/partitioning for hot-key spreading. Metrics: publish/consume rates, unacked count, queue depth, confirm latency, blocked connections. Checklist Queue type chosen (quorum vs classic) per workload. Publisher confirms + unroutable handling. Prefetch/qos tuned; consumers idempotent. Monitoring/alerts on depth, unacked, flow control.

October 18, 2024 · 3510 views