Step 2
Step 2 — Fix HTTP retries and request correlation
2 views
After a network failure, a client cannot know whether the server completed a write. A retry key and an observability request ID solve different problems.
Idempotency-Keymakes one business command converge to one result. A different body under the same key is a409.X-Request-Idconnects one transport attempt to response headers, logs, Kafka, and traces. A retry may have a new request ID while keeping the same business key.
client retry
├─ same Idempotency-Key → same business result
└─ new X-Request-Id → this transport attempt's evidence
Do not automatically retry input or permission errors. Retry only bounded 429, 5xx, and timeout cases with backoff. An unknown write result needs lookup, webhook, or reconciliation rather than a blind second write.
Document and room creation use idempotency keys, while a Kafka producer passes the request ID in a header for consumer logs. Adding correlation without changing the business JSON keeps the contract compatible.
When several scheduler instances can read the same notification, “send, then update sent_at” still races. Before calling the provider, atomically claim with a conditional UPDATE ... WHERE last_attempted_at < today RETURNING id. For features where duplicate delivery and traffic cost are worse than a missed retry, choose an at-most-once daily attempt and keep attempted_at separate from sent_at so an attempt never masquerades as success.
The UI should block duplicate clicks and say “processing” when a command has already been accepted. Errors should describe the user's next action, not an internal exception.