Part 036 — Idempotency Keys, Retry Safety, Timeouts, Deduplication, and Reconciliation

Positioning

Distributed enterprise systems gagal dengan cara yang ambigu.

Contoh:

1. Quote-to-Order service mengirim CreateProductOrder.
2. Product Order berhasil dibuat.
3. Response hilang karena timeout.
4. Caller menganggap gagal.
5. Caller retry.
6. Product Order kedua dibuat.

Kasus serupa dapat menghasilkan:

• duplicate acceptance;
• duplicate approval decision;
• duplicate promotion redemption;
• duplicate capacity reservation;
• duplicate supplier order;
• duplicate Inventory Product;
• duplicate Billing charge;
• dan duplicate customer notification.

Core thesis: timeout bukan bukti kegagalan. Setiap operation dengan side effect harus memiliki logical operation identity, idempotency contract, duplicate-detection retention, retry classification, dan reconciliation strategy. Idempotency adalah domain correctness, bukan sekadar HTTP convenience.

1. Idempotency

An operation is idempotent when repeating the same logical request produces no additional business effect beyond the first successful application.

2. Mathematical Intuition

For operation f:

f(f(x)) = f(x)

In distributed systems, practical idempotency means repeated delivery of one logical command converges to one business outcome.

3. Safe versus Idempotent

Safe

Does not intend to change server state.

Idempotent

May change state, but repeating has no additional effect.

Examples:

• GET is expected safe and idempotent.
• PUT is intended idempotent by HTTP semantics.
• POST is not automatically idempotent.
• DELETE is intended idempotent in resource-state semantics, though repeated responses may differ.

4. Business Idempotency

A business invariant such as:

One Acceptance creates at most one Product Order per conversion group.

5. Transport Idempotency

Repeated HTTP/message delivery returns or recognizes same result.

6. Side-Effect Idempotency

Downstream effect is created once.

7. Idempotency Is End-to-End

Caller-side retry library alone cannot guarantee it.

Each side-effect boundary must participate.

8. Logical Operation

A logical operation is the customer/business intent that may be delivered multiple times.

Examples:

• accept Offer;
• create Product Order;
• reserve promotion;
• submit supplier order;
• activate Billing charge.

9. Operation Identity

Use stable identity:

operationId
idempotencyKey
businessKey
commandId

These concepts may overlap but should be defined.

10. Command ID

Uniquely identifies one command instance.

11. Idempotency Key

Caller-provided or derived key used to deduplicate retries of the same logical operation.

12. Business Key

Domain identity preventing duplicate outcomes.

Examples:

• Acceptance ID;
• Quote Revision + presentation;
• Agreement ID + amendment number;
• Product Order Item + action generation;
• Billing charge component ID.

13. Correlation ID

Connects related operations across a flow.

Not necessarily deduplication identity.

14. Causation ID

Identifies the event/command that caused another message.

15. Message ID

Identifies one delivery message.

Redelivery may use same or different transport message ID depending infrastructure.

16. Idempotency Scope

A key should be scoped by:

• tenant;
• operation;
• client/caller;
• resource;
• and optional time/retention domain.

17. Global Key Risk

A random key reused by another tenant must not collide.

18. Key Composition

Example:

tenantId + operationType + idempotencyKey

19. Server-Generated Business Key

For Quote conversion:

acceptanceId + conversionGroupId

is stronger than a client-random key alone.

20. Client-Generated Key

Useful for network retries before server assigns outcome identity.

21. Both Keys

Use client idempotency key plus domain uniqueness constraints.

22. Key Quality

A good key is:

• stable across retries;
• unique across different logical requests;
• non-sensitive;
• and bounded in size.

23. Key Reuse

Reusing same key for different payload should be rejected.

24. Payload Fingerprint

Store hash of normalized request.

25. Same Key, Same Payload

Return previous result or current processing status.

26. Same Key, Different Payload

Return conflict:

IDEMPOTENCY_KEY_REUSED_WITH_DIFFERENT_REQUEST

27. Canonical Payload

Hashing requires deterministic canonicalization.

28. Canonicalization Risk

Different JSON field order should not create different fingerprints.

29. Sensitive Data in Hash

Hashing does not remove all privacy considerations.

Avoid storing raw secrets unnecessarily.

30. Idempotency Record

Possible fields:

scope
key
requestFingerprint
state
resourceId
responseReference
createdAt
expiresAt

31. Idempotency State

Possible:

• IN_PROGRESS;
• SUCCEEDED;
• FAILED_RETRYABLE;
• FAILED_FINAL;
• UNKNOWN;
• EXPIRED.

32. In-Progress Duplicate

A duplicate request while first is processing can return:

• current operation handle;
• accepted/in-progress;
• or wait for same result.

33. Succeeded Duplicate

Return original result/reference.

34. Failed Duplicate

Behavior depends on failure classification.

35. Unknown Duplicate

Reconcile before creating a new effect.

36. Atomic Reservation of Key

Create/claim idempotency record atomically before side effect.

37. Unique Constraint

Database uniqueness is a strong local safeguard.

38. Transaction Boundary

Ideally persist:

• domain result;
• idempotency result;
• and outbox

in one local transaction.

39. Idempotency Lock

A short local lock/claim may serialize concurrent duplicates.

40. Distributed Lock

Usually not sufficient as sole correctness mechanism.

Locks expire, partitions occur, and downstream effects remain.

41. Unique Business Constraint

Examples:

UNIQUE (tenant_id, acceptance_id, conversion_group_id)

42. Duplicate Request Race

Two identical requests arrive concurrently.

Both must converge on one outcome.

43. Winner

One creates the operation/result.

Others read or wait for winner.

44. Lost Winner Response

Later retry recovers stored result.

45. Retention

Idempotency records need retention long enough to cover:

• retries;
• delayed messages;
• replay;
• and business duplicate risk.

46. Retention by Operation

Examples:

• UI save: hours/days;
• acceptance: years or permanent business uniqueness;
• supplier order: operational lifecycle;
• Billing charge: lifetime of charge identity.

47. Expired Key

If record expires but duplicate can still arrive, domain business key must still prevent duplicate effect.

48. Tombstone

A compact permanent record that an operation/business key was consumed.

49. Garbage Collection

Remove detailed response while retaining uniqueness/evidence where needed.

50. Idempotent Create

A repeated create returns existing resource if same logical operation.

51. Idempotent Update

Use expected version and operation identity.

52. Idempotent Delete

Repeated deletion should converge to deleted/terminated state.

Business deletion often means lifecycle transition, not physical removal.

53. Idempotent Transition

Example:

AcceptOffer

Repeated same acceptance returns same Acceptance ID.

54. Idempotent Notification

More subtle: sending same notification twice is an additional effect.

Use delivery identity/deduplication.

55. Idempotent Email

Store notification instance and recipient/channel key.

56. Idempotent Billing Activation

Use accepted price component identity as business key.

57. Idempotent Inventory Product Creation

Use Product Order Item/outcome identity.

58. Idempotent Supplier Order

Pass stable client order key to supplier if supported.

59. Idempotent Promotion Redemption

Use promotion + customer + redemption intent/business key.

60. Idempotent Reservation

Use source request + resource + reservation purpose.

61. Idempotent Approval Decision

A step/decision slot can have one effective outcome under workflow policy.

62. Retry

Retry repeats an operation after a failure or missing response.

63. Retry Preconditions

Retry only when:

• operation is idempotent;
• failure is classified retryable;
• budget remains;
• and no known success exists.

64. Retryable Failure

Examples:

• temporary unavailable;
• network connection failure before send;
• rate limit;
• transient database conflict;
• and dependency overload.

65. Non-Retryable Failure

Examples:

• invalid request;
• unsupported action;
• authorization denied;
• semantic conflict;
• and permanent business rejection.

66. Unknown Outcome Failure

Examples:

• connection dropped after request body sent;
• timeout after remote processing may have completed;
• and callback lost.

Requires reconciliation, not blind retry.

67. Failure Classification

Use categories:

• TRANSIENT;
• PERMANENT;
• CONFLICT;
• THROTTLED;
• UNKNOWN_OUTCOME;
• CANCELLED;
• and POLICY_REJECTED.

68. Retry Policy

Define:

• eligible failures;
• maximum attempts;
• delay/backoff;
• jitter;
• timeout;
• budget;
• and escalation.

69. Exponential Backoff

Example:

delay = min(base * 2^attempt, maxDelay)

70. Jitter

Randomizes delay to avoid synchronized retry storms.

71. Full Jitter

Random delay between zero and calculated cap.

72. Equal Jitter

Half deterministic, half random.

73. Decorrelated Jitter

Uses prior delay to produce less synchronized sequences.

74. Retry Storm

Many callers retry overloaded dependency, making outage worse.

75. Retry Budget

Limit retries relative to normal traffic.

76. Per-Request Budget

Maximum attempts/time for one operation.

77. Global Budget

Limits retry volume for dependency.

78. Tenant Budget

Prevents one tenant from consuming all retry capacity.

79. Circuit Breaker

Stops requests when dependency is unhealthy.

80. Circuit States

• CLOSED;
• OPEN;
• HALF_OPEN.

81. Circuit Breaker Is Not Retry

It prevents calls and allows recovery probing.

82. Bulkhead

Isolates resources per dependency/tenant/workload.

83. Rate Limit

Controls call rate.

84. Backpressure

Signals producers to reduce work.

85. Queue

Absorbs bursts but can increase latency and stale work.

86. Queue Retry

Message broker redelivery should be bounded and observable.

87. Dead-Letter Queue

Stores messages that cannot be processed automatically.

It is not a final business resolution.

88. Poison Message

Repeated deterministic failure.

Do not retry indefinitely.

89. Retry Ownership

Only one layer should own primary retry for one operation.

90. Retry Multiplication

If API gateway, client, service, workflow, and broker all retry three times:

3 × 3 × 3 × 3 × 3

can explode.

91. Retry Layer Contract

Document:

• which layer retries;
• which errors;
• and how idempotency is propagated.

92. Client Retry

Useful for transport failures.

93. Service Retry

Useful for local dependency call.

94. Workflow Retry

Useful for durable long-running operations.

95. Broker Redelivery

Useful for message processing.

96. Database Transaction Retry

Useful for deadlock/serialization failure.

97. Version Conflict Retry

Automatic retry only if merge/recompute is safe.

98. Retry after Authentication Failure

Usually not without refreshing credentials.

99. Retry after Validation Failure

No.

100. Retry after Rate Limit

Use Retry-After or policy.

101. Timeout

A timeout bounds how long a caller waits.

102. Timeout Types

• connection timeout;
• request timeout;
• response/read timeout;
• operation deadline;
• queue timeout;
• lease timeout;
• and business SLA timeout.

103. Connection Timeout

Connection could not be established.

Often safer to retry because request likely not sent, but transport details matter.

104. Read Timeout

Request may already have been processed.

Outcome can be ambiguous.

105. Operation Deadline

End-to-end deadline propagated across calls.

106. Per-Hop Timeout

Each downstream call must fit within remaining deadline.

107. Deadline Propagation

Pass:

• absolute deadline;
• or remaining duration.

Absolute time requires clock considerations.

108. Timeout Budget

The sum of nested timeouts should not exceed caller deadline.

109. Timeout Hierarchy

Outer timeout should generally exceed inner call timeout plus handling overhead.

110. Infinite Timeout

Dangerous for thread/resource exhaustion.

111. Too-Short Timeout

Creates false failures and duplicate retries.

112. Adaptive Timeout

May use observed latency distribution.

Use carefully to avoid instability.

113. Business Timer versus Technical Timeout

Example:

• approval SLA = business timer;
• HTTP response deadline = technical timeout.

Do not conflate.

114. Timeout Does Not Cancel Remote Work

Unless protocol supports explicit cancellation and remote honors it.

115. Client Cancellation

Caller stops waiting.

Remote operation may continue.

116. Cancellation Token

Can propagate intent, but not guarantee rollback.

117. Orphan Operation

Remote work continues after caller abandons.

Track by operation identity.

118. Ambiguous Outcome

Caller cannot determine whether effect occurred.

119. Ambiguity Sources

• lost response;
• network partition;
• process crash after commit;
• callback loss;
• broker acknowledgement loss;
• and timeout.

120. Three Possible Outcomes

After timeout:

1. operation never arrived;
2. operation arrived and failed;
3. operation succeeded but response was lost.

121. Correct Response to Ambiguity

• query by idempotency/business key;
• inspect operation status;
• reconcile side effects;
• then decide retry or completion.

122. Status Query

Expose:

GET /operations/{operationId}

or lookup by business key.

123. Operation Resource

Long-running operation can be a first-class resource.

124. Operation State

• ACCEPTED;
• RUNNING;
• SUCCEEDED;
• FAILED;
• UNKNOWN;
• CANCEL_REQUESTED;
• CANCELLED.

125. Result Reference

Successful operation points to created/changed domain resource.

126. Polling

Caller polls status.

127. Callback/Webhook

Server pushes result.

128. Polling + Callback

Use callback for efficiency and polling/reconciliation for reliability.

129. Callback Authentication

Validate:

• signature;
• source;
• timestamp;
• and replay protection.

130. Callback Idempotency

Deduplicate callback message.

131. Out-of-Order Callback

Use operation/attempt version.

132. Late Callback

May arrive after retry or supersession.

Do not apply blindly.

133. Reconciliation

Reconciliation compares expected and actual state after uncertainty or eventual consistency.

134. Reconciliation Sources

• operation registry;
• downstream query;
• domain database;
• outbox/inbox;
• external supplier;
• Inventory;
• Billing;
• and message broker metadata.

135. Reconciliation Record

Store:

operation
expectedEffect
observedEffect
classification
resolution
evidence

136. Reconciliation Classifications

• MATCH;
• SUCCEEDED_RESPONSE_LOST;
• NOT_EXECUTED;
• FAILED;
• DUPLICATE;
• PARTIAL;
• CONFLICT;
• and UNRESOLVED.

137. Reconciliation Frequency

Can be:

• immediate after ambiguity;
• scheduled;
• event-triggered;
• and periodic sweep.

138. Reconciliation Window

Allow expected eventual-consistency delay before declaring mismatch.

139. Reconciliation Idempotency

Re-running reconciliation should not create additional business effects.

140. Automatic Resolution

Examples:

• link existing Product Order;
• mark operation succeeded;
• resend missing event;
• and repair projection.

141. Manual Resolution

Needed when:

• multiple possible matches;
• partial external effect;
• inconsistent evidence;
• or irreversible conflict.

142. Manual Resolution Authority

Must be role-controlled and audited.

143. No Blind Repair

Do not create a replacement effect before confirming old effect absent.

144. Outbox Pattern

Local transaction persists:

• domain change;
• and event intent.

A publisher sends later.

145. Outbox Duplicate Publish

Publisher may send event more than once.

Consumers must deduplicate.

146. Inbox Pattern

Consumer records processed message ID/operation identity.

147. Inbox Transaction

Persist:

• deduplication marker;
• and local domain effect

in one transaction.

148. Broker Exactly-Once

Broker exactly-once features do not guarantee end-to-end exactly-once across database, APIs, and external systems.

149. At-Least-Once Delivery

Expect duplicates.

150. At-Most-Once Delivery

Avoids duplicates but can lose messages.

151. Effectively-Once

Achieved through:

• at-least-once delivery;
• idempotent effects;
• uniqueness;
• and reconciliation.

152. Transactional Messaging

Outbox/inbox aligns local state and messages.

153. Dual Write

Writing database and publishing message separately risks inconsistency.

154. Dual-Write Failure Modes

• DB commit, event lost;
• event published, DB rolled back;
• and duplicate event.

155. Distributed Transaction

Two-phase commit may be unsuitable across modern services/external providers.

156. Saga and Idempotency

Every saga step and compensation should be idempotent.

157. Compensation Idempotency

Repeated compensation converges to one remedial outcome.

158. Compensation Ambiguity

Compensation can also time out and require reconciliation.

159. Reservation Idempotency

Repeated release should not release another owner’s reservation.

160. Versioned Ownership Token

Reservation/lease operations should include ownership/version token.

161. Fencing Token

Monotonic token prevents stale lock holder from writing.

162. Lease Expiry Risk

Old holder may continue after lease expiry.

Use fencing where shared resource requires.

163. Optimistic Concurrency

Protects lost updates using expected version.

164. Idempotency versus Optimistic Concurrency

Idempotency

Deduplicates same logical operation.

Optimistic concurrency

Rejects stale operation based on old state.

Often both are needed.

165. Duplicate versus Conflict

A duplicate same request can return prior success.

A different request against stale version should return conflict.

166. Compare-and-Set

Conditional update ensures state transition happens once.

167. Unique Constraint

Prevents duplicate resource/effect locally.

168. State-Transition Guard

Example:

PRESENTED -> ACCEPTED

only once.

169. Natural Idempotency

Setting state to a value can be naturally idempotent if no extra effects are repeated.

170. Hidden Non-Idempotent Side Effect

Even if state already ACCEPTED, repeated handler may:

• create another Order;
• send another email;
• and redeem promotion twice.

Each side effect needs its own identity.

171. Idempotency Tree

One command may trigger many effects.

Each effect needs deterministic child key.

Example:

Acceptance ACC-1
├── Agreement create key ACC-1:agreement
├── Order group A key ACC-1:order:A
├── Order group B key ACC-1:order:B
└── Notification key ACC-1:confirmation:customer

172. Derived Child Key

Stable derivation prevents duplicate child effects across replay.

173. Fan-Out Idempotency

Every child message gets stable logical identity.

174. Batch Idempotency

Batch retry should not repeat already successful items.

175. Per-Item Result

Return/store:

• succeeded;
• failed;
• unknown;
• and duplicate.

176. Partial Batch Retry

Retry only unresolved/retryable items.

177. Bulk API Idempotency

Batch key plus per-item keys.

178. Ordering

Idempotency does not guarantee message order.

179. Sequence Number

Use aggregate/item sequence for ordering and stale-event detection.

180. Gap Detection

Consumer may detect missing sequence.

181. Out-of-Order Event

Ignore, buffer, or reconcile according to projection semantics.

182. Event Version

Include aggregate version.

183. Monotonic State

Terminal facts should not be overwritten by older events.

184. Clock Skew

Do not order distributed business events only by timestamps.

185. Observability

Every operation should log/trace:

• operation ID;
• idempotency key hash/reference;
• business key;
• attempt;
• and result.

186. Sensitive Key

Do not put secrets or personal data in idempotency keys.

187. Trace Correlation

Propagate correlation and causation IDs.

188. Attempt Metric

Track attempts per logical operation.

189. Duplicate Metric

Count duplicate requests and deduplicated effects.

190. Timeout Metric

Break down by:

• dependency;
• operation;
• and timeout type.

191. Ambiguous Outcome Metric

Track unresolved and resolution time.

192. Retry Metric

• retry count;
• success after retry;
• exhausted retries;
• and storm suppression.

193. Reconciliation Metric

• mismatches;
• recovered existing effect;
• duplicates;
• and manual resolution.

194. Idempotency SLI

Examples:

• zero duplicate Product Orders per Acceptance/group;
• zero duplicate Billing charge per accepted price component;
• all ambiguous outcomes reconciled within target;
• and all repeated acceptance requests return same effective outcome.

Internal targets must be verified.

195. Alerting

Alert on:

• duplicate uniqueness violations;
• rising timeout rate;
• unknown outcomes above threshold;
• retry storm;
• and reconciliation backlog.

196. Stuck Operation

Examples:

• IN_PROGRESS beyond deadline;
• UNKNOWN unresolved;
• retry scheduled but timer missing;
• and callback received without operation.

197. Operation Reaper

Finds abandoned/stuck operation records.

198. Safe Reaper Action

• reconcile;
• mark manual review;
• or retry only after absence confirmed.

199. Idempotency Store Availability

If store unavailable:

• fail closed for high-risk effects;
• or use domain uniqueness as fallback.

Do not silently disable deduplication.

200. Idempotency Store Partition

Scope storage by tenant/operation and scale for hot keys.

201. Hot Key

Many retries on same key can overload one partition.

202. Thundering Herd on Completion

Many duplicate callers wait for same operation.

Use:

• status resource;
• bounded wait;
• and notification.

203. Cache

A cache alone is insufficient for authoritative idempotency because it can evict.

204. Durable Store

Use durable database/log for high-value operations.

205. Response Storage

Options:

• full response;
• resource reference;
• status reference;
• and response hash.

206. Large Response

Store stable resource ID and reconstruct response safely.

207. Schema Evolution

Historical idempotency result may use old response schema.

208. API Contract

Support versioned result interpretation.

209. HTTP Idempotency Key

Typical request header:

Idempotency-Key: <opaque-key>

Exact API governance must be verified internally.

210. HTTP Response

Possible:

• original success;
• 202 Accepted with operation location;
• 409 Conflict for same key/different request;
• and retryable service error.

211. Retry-After

Use for throttling or temporary unavailable where meaningful.

212. HTTP Timeout Response

Client may receive no response at all.

Server still needs operation lookup.

213. Async Operation Pattern

Response:

202 Accepted
Location: /operations/OP-123

214. Event Command Envelope

Possible fields:

messageId
commandId
idempotencyKey
correlationId
causationId
tenantId
aggregateId
expectedVersion
payload

215. Consumer Envelope Validation

Check:

• required identity;
• schema version;
• tenant;
• and signature/authentication.

216. Deduplication Key Selection

Prefer logical command/effect ID over raw broker message ID when redelivery can change transport ID.

217. Security

Attackers can exploit idempotency to:

• probe prior requests;
• reuse another user’s key;
• and cause storage exhaustion.

218. Authorization on Duplicate

A duplicate lookup still requires current/request-scoped authorization.

219. Cross-Tenant Key Isolation

Mandatory.

220. Key Guessing

Keys should be high entropy or scoped to authenticated owner.

221. Storage Exhaustion

Rate-limit creation of idempotency records.

222. Response Leakage

Do not return original result to another unauthorized caller who guesses key.

223. Replay Protection

Signed callbacks/webhooks should include:

• timestamp;
• nonce/message ID;
• and signature.

224. Old Replay

Reject outside allowed time window or if already consumed.

225. Idempotency Testing

Test:

• same key same payload sequentially;
• same key concurrently;
• same key different payload;
• and retry after response loss.

226. Crash Testing

Crash:

• before key claim;
• after key claim;
• after side effect;
• before response;
• after DB commit;
• before outbox publish.

227. Network Fault Testing

Inject:

• connection reset;
• delayed response;
• duplicate response;
• lost callback;
• and partition.

228. Broker Testing

Inject:

• duplicate delivery;
• redelivery after commit;
• out-of-order delivery;
• and poison message.

229. Timeout Testing

Test:

• just before timeout;
• at boundary;
• after remote success;
• and nested deadline exhaustion.

230. Retry Testing

Verify:

• bounded attempts;
• backoff;
• jitter;
• and retry budget.

231. Reconciliation Testing

Create expected/actual mismatches and verify safe recovery.

232. Property-Based Testing

Properties:

• one logical create yields at most one business resource;
• duplicate completion yields one terminal effect;
• same key/different payload never reuses result;
• retry never occurs after confirmed success;
• and unknown outcome is not classified as failure without evidence.

233. Chaos Testing

Useful for non-production or controlled environments:

• kill process after commit;
• delay broker acknowledgement;
• drop response;
• and duplicate callbacks.

234. Idempotency Smells

• random key generated on each retry;
• cache-only deduplication;
• and no payload fingerprint.

235. Retry Smells

• retry all exceptions;
• fixed immediate retries;
• multiple layers retry;
• and no budget.

236. Timeout Smells

• one timeout for every operation;
• timeout interpreted as rollback;
• and remote work not queryable.

237. Reconciliation Smells

• manual SQL as only recovery;
• no operation identity;
• and no downstream lookup key.

238. Messaging Smells

• exactly-once assumption;
• event handler performs non-idempotent side effects;
• and message ID not persisted atomically.

239. API Smells

• POST create without idempotency;
• duplicate returns generic 500;
• and same key/different payload accepted.

240. Anti-Patterns

Retry on Timeout Blindly

Creates duplicate effects.

Idempotency Key without Domain Uniqueness

Expired record can still allow duplicate business outcome.

Distributed Lock as Idempotency

Lock loss/expiry does not prove effect absence.

Cache as Source of Truth

Eviction re-enables duplicate.

Exactly-Once Marketing Assumption

External side effects remain outside guarantee.

Timeout Equals Cancel

Remote work may continue.

DLQ Equals Resolution

Business inconsistency remains.

Retry at Every Layer

Traffic amplification.

241. Idempotency Contract Template

## Operation

## Logical Intent

## Idempotency Scope

## Client Key

## Business Key

## Request Fingerprint

## Atomic Claim

## Result Storage

## Duplicate Behavior

## Same-Key/Different-Payload Behavior

## Retention / Tombstone

## Downstream Child Keys

## Reconciliation

## Security

242. Retry Policy Template

Operation:
Retry owner:
Retryable failures:
Non-retryable failures:
Unknown-outcome behavior:
Max attempts:
Backoff:
Jitter:
Per-request budget:
Global/tenant budget:
Circuit breaker:
Escalation:

243. Timeout Policy Template

Operation:
Connection timeout:
Request/read timeout:
End-to-end deadline:
Downstream budget:
Cancellation behavior:
Unknown-outcome classification:
Status query:
Reconciliation window:

244. Operation Record Template

Operation ID:
Tenant/scope:
Command/business key:
Idempotency key:
Request fingerprint:
State:
Attempt count:
Resource/result reference:
Created/updated:
Deadline:
Unknown outcome:
Reconciliation:

245. Attempt Record Template

Attempt ID:
Operation:
Started at:
Endpoint/participant:
Command/message ID:
Timeout:
Result:
Failure class:
Retry scheduled:
External reference:

246. Reconciliation Record Template

Operation:
Expected effect:
Observed effects:
Evidence sources:
Classification:
Duplicate/conflict:
Resolution:
Resolved by:
Resolved at:

247. Child-Key Template

Parent operation:
Effect type:
Stable effect scope:
Derived key:
Target system:
Uniqueness guarantee:

248. Idempotency Invariants

Representative invariants:

• same logical Acceptance has one effective Acceptance record;
• same Acceptance/group has at most one Product Order;
• same Order Item action creates at most one intended Inventory outcome;
• same accepted charge component creates at most one Billing charge identity;
• same key with different request is rejected;
• confirmed success is never retried as a new logical operation;
• unknown outcome is reconciled before duplicate side effect;
• and deduplication state is persisted atomically with local effect where possible.

249. Worked Example: Offer Acceptance

Customer double-clicks Accept.

Both requests use same idempotency key.

One Acceptance is committed.

Second request returns same Acceptance ID.

250. Worked Example: Concurrent Acceptance

Two identical requests arrive at the same time.

Unique key/conditional transition allows one winner.

The other observes completed result.

251. Worked Example: Key Reuse Conflict

Client sends same key with different Proposal ID.

Server rejects with idempotency conflict.

252. Worked Example: Product Order Timeout

Order create succeeds, response is lost.

Caller queries by:

• Acceptance ID;
• conversion group.

Existing Product Order is linked.

253. Worked Example: Supplier Order Timeout

Supplier supports client order reference.

Retry uses same reference.

Supplier returns existing order rather than creating duplicate.

254. Worked Example: Supplier without Idempotency API

Caller stores operation and queries supplier using searchable external reference.

If no reliable lookup exists, use conservative manual reconciliation rather than blind retry for high-value effects.

255. Worked Example: Billing Activation

Accepted recurring component has stable charge ID.

Repeated Billing activation command converges on one Billing charge.

256. Worked Example: Inventory Product Creation

ADD Order Item emits planned Product identity.

Inventory enforces unique source Order Item/outcome key.

Duplicate callback does not create second Product.

257. Worked Example: Promotion Redemption

Promotion reservation succeeds but response is lost.

Retry with same redemption key returns original reservation.

258. Worked Example: Retry Storm

Dependency outage triggers thousands of failures.

Circuit breaker opens, retries use jitter and budget, and queued work is rate-limited.

259. Worked Example: Nested Retry Multiplication

Gateway, service, workflow, and SDK each retry.

Policy removes overlapping retries and establishes one durable owner.

260. Worked Example: Unknown Callback

Completion callback arrives after operation marked timeout.

Operation reconciles to success and cancels scheduled retry.

261. Worked Example: Compensation Ambiguity

Cancellation command times out.

System queries downstream before issuing another cancel or compensating related resources.

262. Worked Example: Outbox Crash

Domain commit and outbox record succeed.

Process crashes before publish.

Outbox worker publishes later, possibly more than once.

Consumer inbox deduplicates.

263. Worked Example: Consumer Crash

Consumer applies DB change and inbox marker in one transaction.

Crash before broker acknowledgement causes redelivery.

Inbox recognizes duplicate.

264. Worked Example: Batch Site Creation

1,000 site items submitted.

800 succeed, 150 fail retryably, 50 unknown.

Retry only 150 after policy; reconcile 50 first.

265. Worked Example: Idempotency Record Expiry

Detailed HTTP response record expired.

Permanent domain uniqueness on Acceptance still prevents second Product Order.

266. Worked Example: Fencing Token

A lease holder resumes after pause with old token.

Resource rejects stale token because newer holder has higher fencing number.

267. Senior Engineer Operating Model

Identify logical operations

Not merely requests/messages.

Use both idempotency and business uniqueness

Client key plus domain invariant.

Store request fingerprint

Reject key reuse with changed intent.

Treat timeout as ambiguity

Reconcile before retry.

Assign retry ownership

Prevent multiplication.

Bound retries

Backoff, jitter, budgets, circuit breakers.

Make child effects idempotent

Order, Inventory, Billing, supplier, notification.

Persist outbox/inbox atomically

Expect duplicate delivery.

Operate reconciliation

Unknown outcomes and duplicate effects are production data.

Test crash points

Especially after commit and before response.

268. Internal Verification Checklist

Identity and keys

• Which operations require idempotency?
• What client key format/scope exists?
• What domain business keys prevent duplicate outcomes?
• Are child side-effect keys derived deterministically?

Atomicity and storage

• Where are idempotency records stored?
• Is key claim atomic?
• Is result/outbox persisted in same transaction?
• How long are records/tombstones retained?

Duplicate behavior

• What happens for same key/same payload?
• What happens for same key/different payload?
• What happens while first operation is still running?
• Is authorization rechecked on duplicate lookup?

Retry

• Which layer owns retry?
• Which failures are retryable, permanent, or unknown?
• What backoff, jitter, attempt limit, and budget apply?
• Are circuit breakers and bulkheads used?

Timeouts

• Which timeout types are defined?
• Are deadlines propagated?
• Does timeout cancel remote work?
• Is operation status queryable?

Ambiguous outcomes

• Which operations can return UNKNOWN?
• What downstream business key supports lookup?
• How quickly is reconciliation triggered?
• When is manual review required?

Messaging

• Are outbox and inbox used?
• Are consumers idempotent?
• Are events sequenced/versioned where needed?
• Is “exactly once” assumed incorrectly?

Critical domains

• Can Acceptance duplicate?
• Can Product Order, Agreement, supplier order, Inventory Product, Billing charge, promotion redemption, or notification duplicate?
• What uniqueness constraint exists for each?
• What historical incidents reveal missing idempotency?

269. Practical Exercises

Exercise 1 — Logical operation inventory

List every side-effecting operation in Quote-to-Cash and its business key.

Exercise 2 — Idempotency design

Design same-key/same-payload, concurrent duplicate, and changed-payload behavior.

Exercise 3 — Retry ownership

Map all current retry layers and eliminate multiplication.

Exercise 4 — Timeout classification

Classify connection, read, deadline, business timer, and ambiguous timeout outcomes.

Exercise 5 — Reconciliation

Design downstream lookup and repair for Product Order, supplier, Inventory, and Billing.

Exercise 6 — Crash matrix

Test failures before/after commit, outbox publish, response, and acknowledgement.

270. Part Completion Checklist

You are done if you can:

• define logical operation identity;
• distinguish idempotency key, business key, command ID, correlation ID, and message ID;
• atomically claim and persist idempotent outcomes;
• reject same-key/different-payload reuse;
• define retention and tombstones;
• classify retryable, permanent, and unknown failures;
• configure bounded retries and deadline propagation;
• reconcile ambiguous outcomes before retry;
• make child side effects idempotent;
• use outbox/inbox under at-least-once delivery;
• and create an internal idempotency/retry verification backlog.

271. Key Takeaways

1. Timeout is not proof of failure.
2. Idempotency protects logical operations, not only HTTP requests.
3. Business uniqueness complements idempotency records.
4. Same key with different payload must be rejected.
5. Every child side effect needs stable identity.
6. Retry only classified retryable operations.
7. Unknown outcomes require reconciliation.
8. At-least-once delivery plus idempotency provides effectively-once effects.
9. Retry multiplication can amplify outages.
10. Internal CSG idempotency and timeout contracts must be verified.

272. References

Conceptual baseline:

• HTTP safe/idempotent method semantics and idempotency-key patterns.
• Distributed systems retries, exponential backoff, jitter, deadlines, circuit breakers, bulkheads, and backpressure.
• Transactional outbox, inbox/deduplication, at-least-once delivery, sagas, and effectively-once effects.
• Domain-Driven Design commands, business identities, uniqueness invariants, and reconciliation.
• Enterprise Quote-to-Order, Inventory, supplier, Billing, and notification side-effect patterns.

These references do not define internal CSG retry libraries, idempotency stores, or distributed-operation contracts.