Backend

Producers acks=all, min.insync.replicas>=2, idempotent producer on; enable transactions for exactly-once pipelines. Tune batch.size/linger.ms for throughput; cap max.in.flight.requests.per.connection for ordering. Handle retries with backoff; surface delivery errors. Consumers enable.auto.commit=false; commit after processing; DLT for poison messages. Size max.poll.interval.ms to work time; bound max.poll.records. Isolate heavy work in worker pool; keep poll loop fast. Topics & brokers RF ≥ 3; clean-up policy fit (delete vs compact); segment/retention sized to storage. Monitor ISR, under-replicated partitions, controller changes, request latency, disk usage. Throttle large produce/fetch; use quotas per client if needed. Checklist Producers idempotent, acks=all, MISR set. Consumers manual commit + DLT. RF/retention sized; ISR/URP monitored. Alerts on broker latency/disk/replication health.

Performance notes Fastify: schema-driven, AJV validation, low-overhead routing; typically better RPS and lower p99s. Express: mature ecosystem; middleware can add overhead; great for quick prototypes. Bench considerations Compare with same validation/parsing; disable unnecessary middleware in Express. Test under keep-alive and HTTP/1.1 & HTTP/2 where applicable. Measure event loop lag and heap; not just RPS. Migration tips Start new services with Fastify; for existing Express apps, migrate edge routes first. Replace middleware with hooks/plugins; map Express middlewares to Fastify equivalents. Validate payloads with JSON schema for speed and safety. Operational tips Use pino (Fastify default) for structured logs; avoid console log overhead. Keep plugin count minimal; watch async hooks cost. Load test with realistic payload sizes; profile hotspots before/after migration. Takeaway: Fastify wins on raw throughput and structured DX; Express still fine for smaller apps, but performance-critical services benefit from Fastify’s lower overhead.

Approaches Token bucket for burst+steady control; sliding window for fairness. Enforce at edge (gateway/ingress) plus app-level for per-tenant safety. Spring implementation Use filters/interceptors with Redis/Lua for atomic buckets. Key by tenant/user/IP; return 429 with Retry-After. Expose metrics per key and rule; alert on near-capacity. Considerations Separate auth failures from rate limits; avoid blocking login endpoints too aggressively. Keep rule configs dynamic; hot-reload from config store. Combine with circuit breakers/timeouts for upstream dependencies. Checklist Edge and app-level limits defined. Redis-based atomic counters/buckets with TTL. Metrics + logs for limit decisions; alerts in place.

Principles Keep goroutine count bounded; size pools to CPU/core and downstream QPS. Apply backpressure: bounded channels + select with default to shed load early. Context everywhere: cancel on timeouts/parent cancellation; close resources. Prefer immutability; minimize shared state. Use channels for coordination. Patterns Worker pool with buffered jobs; fan-out/fan-in via contexts. Rate limit with time.Ticker or golang.org/x/time/rate. Semaphore via buffered channel for limited resources (DB, disk, external API). Sync cheatsheet sync.Mutex for critical sections; avoid long hold times. sync.WaitGroup for bounded concurrent tasks; errgroup for cancellation on first error. sync.Map only for high-concurrency, write-light cases; prefer map+mutex otherwise. Instrument & guard pprof: net/http/pprof + CPU/mem profiles in staging under load. Trace blocking: GODEBUG=schedtrace=1000,scavtrace=1 when diagnosing. Metrics: goroutines, GC pause, allocations, queue depth, worker utilization. Checklist Context per request; timeouts set at ingress. Bounded goroutines; pools sized and observable. Backpressure on queues; drop/timeout strategy defined. pprof/metrics enabled in non-prod and behind auth in prod. Load tests for saturation behavior and graceful degradation.

Workers & balancing Define queue priorities; dedicate workers per queue (emails, webhooks, default). Use balance strategies (simple, auto) and cap max processes per supervisor. Reliability Set retry/backoff per job; push non-idempotent tasks carefully. Configure timeout and retry_after (keep retry_after > max job time). Use Redis with persistence; enable horizon:supervisors monitors. Observability Horizon dashboard: throughput, runtime, failures, retries. Alert on rising failures and long runtimes; log payload/context for failed jobs. Prune failed jobs with retention policy; send to DLQ when needed. Deployment Restart Horizon on deploy to pick up code; use horizon:terminate. Ensure supervisor/systemd restarts Horizon if it dies. Checklist Queues prioritized; supervisors sized. Retries/backoff and timeouts set; DLQ plan. Monitoring/alerts configured; failed job retention in place.

Identity & access Enforce strong auth (OIDC/JWT); short-lived tokens + refresh; audience/issuer checks. Fine-grained authz (RBAC/ABAC); deny-by-default; rate-limit per identity. Input & data Validate/normalize input; reject oversized bodies; JSON schema where possible. Output encode; avoid reflecting raw user data; paginate results. Store secrets in vault/KMS; rotate keys; never log secrets/tokens. Transport & headers TLS everywhere; HSTS; modern ciphers. Security headers: Content-Security-Policy, X-Content-Type-Options=nosniff, X-Frame-Options=DENY, Referrer-Policy. Abuse protection Rate limit + burst control; CAPTCHA/step-up for sensitive actions. Bot detection where relevant; geo/IP allow/deny for admin surfaces. Observability Structured audit logs with identity, action, resource, result; avoid PII spill. Alerts on auth failures, unusual rate spikes, and 5xx anomalies. Checklist AuthZ enforced; least privilege. Input validated; size limits set. TLS + security headers applied. Rate limits + abuse controls configured. Secrets vaulted and rotated.

Core settings max.poll.interval.ms sized to processing time; max.poll.records to batch size. fetch.min.bytes/fetch.max.wait.ms to trade latency vs throughput. enable.auto.commit=false; commit sync/async after processing batch. Concurrency Prefer multiple consumer instances over massive max.poll.records. For CPU-bound steps, hand off to bounded executor; avoid blocking poll thread. Ordering & retries Keep partition affinity when ordering matters; use DLT for poison messages. Backoff with jitter on retries; limit attempts per message. Observability Metrics: lag per partition, commit latency, rebalances, processing time, error rates. Log offsets and partition for errors; trace batch sizes. Checklist Poll loop never blocks; work delegated to bounded pool. Commits after successful processing; DLT in place. Lag and rebalance metrics monitored.

Designing a REST API that is intuitive, maintainable, and scalable requires following established best practices. Here’s a comprehensive guide. 1. Use Nouns, Not Verbs Good GET /users GET /users/123 POST /users PUT /users/123 DELETE /users/123 Bad GET /getUsers GET /getUserById POST /createUser PUT /updateUser DELETE /deleteUser 2. Use Plural Nouns Good GET /users GET /orders GET /products Bad GET /user GET /order GET /product 3. Use HTTP Methods Correctly // GET - Retrieve resources GET /users // Get all users GET /users/123 // Get specific user // POST - Create new resources POST /users // Create new user // PUT - Update entire resource PUT /users/123 // Replace user // PATCH - Partial update PATCH /users/123 // Update specific fields // DELETE - Remove resources DELETE /users/123 // Delete user 4. Use Proper HTTP Status Codes // Success 200 OK // Successful GET, PUT, PATCH 201 Created // Successful POST 204 No Content // Successful DELETE // Client Errors 400 Bad Request // Invalid request 401 Unauthorized // Authentication required 403 Forbidden // Insufficient permissions 404 Not Found // Resource doesn't exist 409 Conflict // Resource conflict // Server Errors 500 Internal Server Error 503 Service Unavailable 5. Consistent Response Format { "data": { "id": 123, "name": "John Doe", "email": "[email protected]" }, "meta": { "timestamp": "2024-11-15T10:00:00Z" } } Error Response Format { "error": { "code": "VALIDATION_ERROR", "message": "Invalid input data", "details": [ { "field": "email", "message": "Invalid email format" } ] } } 6. Versioning URL Versioning /api/v1/users /api/v2/users Header Versioning Accept: application/vnd.api+json;version=1 7. Filtering, Sorting, Pagination GET /users?page=1&limit=20 GET /users?sort=name&order=asc GET /users?status=active&role=admin GET /users?search=john 8. Nested Resources GET /users/123/posts POST /users/123/posts GET /users/123/posts/456 PUT /users/123/posts/456 DELETE /users/123/posts/456 9. Use HTTPS Always use HTTPS in production to encrypt data in transit. ...

Capturing safely Expose /debug/pprof behind auth/VPN; or run curl -sK -H "Authorization: Bearer ...". CPU profile: go tool pprof http://host/debug/pprof/profile?seconds=30. Heap profile: .../heap; Goroutines: .../goroutine?debug=2. For containers: kubectl port-forward then grab profiles; avoid prod CPU throttle when profiling. Reading CPU profiles Look at flat vs. cumulative time; identify hot functions. Flamegraph: go tool pprof -http=:8081 cpu.pprof. Check GC activity and syscalls; watch for mutex contention. Reading heap profiles Compare live allocations vs. in-use objects; watch large []byte and map growth. Look for leaks via rising heap over time; diff profiles between runs. Goroutine dumps Spot leaked goroutines (blocked on channel/lock/I/O). Common culprits: missing cancel, unbounded worker creation, stuck time.After. Best practices Add pprof only when needed in prod; default on in staging. Sample under load close to real traffic. Keep artifacts: store profiles with build SHA + timestamp; compare after releases. Combine with metrics (alloc rate, GC pauses, goroutines) to validate fixes.