Part 035 — Dependency DAGs, Execution Coordination, Milestones, Barriers, and Orchestration

Positioning

Fulfillment enterprise hampir selalu berbentuk graph.

Satu Product Order dapat memerlukan:

• site validation;
• capacity reservation;
• supplier order;
• resource allocation;
• device shipment;
• field appointment;
• installation;
• activation;
• verification;
• Inventory update;
• Billing activation;
• dan customer notification.

Jika hubungan antar-work item hanya disimpan sebagai urutan angka atau status global, sistem akan sulit:

• menjalankan pekerjaan secara paralel;
• memahami critical path;
• mengisolasi fallout;
• melakukan retry;
• melakukan replan;
• dan menjelaskan mengapa satu Order tertahan.

Core thesis: orchestration harus mengeksekusi graph dependency yang versioned dan deterministic. Dependency, barrier, milestone, atomicity group, retry boundary, dan compensation boundary harus first-class, bukan tersembunyi di procedural code atau message-order assumptions.

1. Dependency Graph

A Dependency Graph models relationships among executable fulfillment units.

Nodes can represent:

• Product Order Items;
• Service Order Items;
• Resource Order Items;
• supplier work;
• manual tasks;
• appointments;
• reservations;
• milestones;
• and verification gates.

Edges represent execution constraints or semantic dependencies.

2. Directed Graph

Most execution dependencies are directional.

Example:

Reserve Capacity -> Install Access -> Activate Service

3. Directed Acyclic Graph

A DAG has no directed cycle.

DAGs support:

• topological ordering;
• parallel execution;
• critical-path analysis;
• and deterministic dependency resolution.

4. Why Cycles Are Dangerous

Cycle example:

A waits for B
B waits for C
C waits for A

The Order cannot progress without special coordination semantics.

5. Cycle Detection

Run before Plan publication and after any replan.

Algorithms:

• depth-first search;
• Kahn topological sort;
• strongly connected components.

6. Strongly Connected Component

A non-trivial strongly connected component indicates a cycle.

It should be:

• rejected;
• transformed into explicit coordinated group;
• or sent to manual design.

7. Dependency Node

A node should have:

nodeId
nodeType
sourceOrderItem
owner
action
state
expectedOutcome

8. Dependency Edge

An edge should have:

edgeId
fromNode
toNode
dependencyType
condition
failurePolicy
timingConstraint
sourceRule

9. Dependency Identity

Give every dependency stable identity for:

• diagnostics;
• updates;
• replan;
• and audit.

10. Dependency Source

Possible sources:

• decomposition rule;
• catalog relationship;
• manually approved plan;
• supplier constraint;
• and runtime-discovered dependency.

11. Source Version

Retain rule/plan version that created the dependency.

12. Dependency Types

Common types:

• START_AFTER;
• COMPLETE_AFTER;
• START_BEFORE;
• COMPLETE_BEFORE;
• START_TOGETHER;
• COMPLETE_TOGETHER;
• MUTUALLY_EXCLUSIVE;
• REQUIRES_SUCCESS;
• REQUIRES_TERMINAL;
• and CONDITIONAL.

13. START_AFTER

Node B may start after A reaches configured milestone/state.

14. COMPLETE_AFTER

B may start earlier but cannot complete before A.

15. START_BEFORE

A must start before B.

Usually normalize into directional relation.

16. COMPLETE_BEFORE

A must complete before B completes.

17. START_TOGETHER

Nodes should begin within an allowed synchronization window.

18. COMPLETE_TOGETHER

Nodes should complete within a synchronization window.

19. MUTUALLY_EXCLUSIVE

Only one branch can execute.

20. REQUIRES_SUCCESS

Dependent node requires successful predecessor outcome.

21. REQUIRES_TERMINAL

Dependent node waits for predecessor terminal outcome, regardless of success.

Useful for cleanup or post-processing.

22. CONDITIONAL Dependency

Edge applies only if predicate is true.

Example:

replaceRouter = true

23. Dependency Condition

Conditions should be:

• machine-readable;
• deterministic;
• versioned;
• and evaluated against explicit context.

24. Dependency State

An edge can be:

• INACTIVE;
• WAITING;
• SATISFIED;
• FAILED;
• WAIVED;
• or SUPERSEDED.

25. Dependency Satisfaction

A dependency is satisfied when its exact condition and predecessor outcome hold.

26. Dependency Failure

Possible when:

• predecessor failed;
• deadline passed;
• condition cannot be evaluated;
• or predecessor was cancelled.

27. Failure Propagation Policy

Possible policies:

• BLOCK_DEPENDENT;
• CANCEL_DEPENDENT;
• FAIL_DEPENDENT;
• CONTINUE_DEGRADED;
• CHOOSE_ALTERNATIVE;
• and MANUAL_REVIEW.

28. Default Failure Policy

Avoid one global default for all dependencies.

29. Dependency Waiver

A waiver may allow dependent execution despite unmet dependency.

Requires:

• authority;
• reason;
• scope;
• validity;
• and audit.

30. Non-Waivable Dependency

Examples:

• safety;
• regulatory approval;
• mandatory capacity reservation;
• and source Product existence.

31. Dependency Graph Scope

Possible scopes:

• one Order;
• one Order Item;
• one fulfillment domain;
• one delivery wave;
• or cross-Order.

32. Cross-Order Dependency

Example:

• one access Order must complete before several service Orders.

Cross-Order coordination should be explicit.

33. Cross-Tenant Dependency

Generally prohibited.

34. Cross-Domain Dependency

Examples:

• network access before CPE activation;
• CPE activation before monitoring;
• service activation before Billing.

35. Cross-Plan Dependency

If federated Plans exist, coordinator owns cross-fragment dependency.

36. Dependency Graph Version

A published graph should be immutable.

Replan creates a new graph/version.

37. Graph Checksum

Can help detect accidental mutation.

38. Graph Manifest

Possible fields:

graphId
planVersion
nodeVersions
edgeVersions
ruleVersions
checksum
publishedAt

39. Topological Order

A valid DAG can be topologically sorted.

40. Topological Order Is Not Execution Order

Execution may run independent nodes in parallel.

Topological order is one valid dependency-respecting ordering.

41. Ready Set

At any point, Ready Set contains nodes whose prerequisites are satisfied.

42. Frontier

The current executable frontier of the DAG.

43. Parallel Execution

Independent nodes can execute concurrently.

44. Parallelism Limit

Bound by:

• downstream capacity;
• tenant quota;
• supplier limits;
• rate limits;
• and operational policy.

45. Concurrency Window

A scheduler may execute up to N nodes concurrently for a scope.

46. Fairness

Avoid one large Order starving smaller Orders.

47. Priority

Priority may consider:

• customer commitment;
• due date;
• severity;
• dependency criticality;
• and business class.

48. Priority Is Not Dependency

High priority cannot violate prerequisite constraints.

49. Critical Path

The longest duration path determines earliest possible completion.

50. Critical Path Node

Delay on a critical-path node delays final completion.

51. Slack

Slack is delay tolerance without affecting final milestone.

52. Criticality Recalculation

Recompute after:

• duration change;
• failure;
• replan;
• and supplier delay.

53. Duration Estimate

Every schedulable node may have:

• expected duration;
• confidence;
• and source.

54. Duration Distribution

For uncertain work, use ranges or probabilistic estimates.

55. Earliest Start

Derived from predecessor completion and calendars.

56. Latest Start

Derived from required completion date and downstream durations.

57. Earliest Finish

Earliest start plus expected duration.

58. Latest Finish

Latest time without violating final deadline.

59. Deadline

Can be:

• customer-requested;
• provider-committed;
• regulatory;
• or internal SLA.

60. Timezone and Calendar

All scheduling must use explicit:

• timezone;
• business calendar;
• holiday calendar;
• and maintenance windows.

61. Milestone

A Milestone is a significant achieved fact.

Examples:

• design approved;
• capacity reserved;
• site ready;
• installation complete;
• service active;
• Billing active.

62. Milestone versus Node

A milestone may be:

• emitted by one node;
• aggregated from several nodes;
• or a zero-duration graph node.

63. Milestone Identity

Store:

• milestone ID;
• type;
• scope;
• achievedAt;
• source;
• and evidence.

64. Milestone Gate

A later branch may wait for milestone rather than one specific task.

65. Milestone Aggregation

Example:

SITE_READY
requires:
- power available
- access approved
- room prepared

66. Milestone Correction

Do not delete historical milestone.

Create correction/supersession evidence.

67. Barrier

A Barrier waits for multiple predecessor conditions.

68. AND Barrier

All predecessors must satisfy.

69. OR Barrier

Any qualifying predecessor may satisfy.

70. N-of-M Barrier

A quorum of predecessors is sufficient.

71. Barrier Identity

First-class identity helps explain waiting.

72. Barrier Timeout

A barrier may time out and trigger:

• fallback;
• cancellation;
• manual review;
• or replan.

73. Barrier Release

Record which conditions released the barrier.

74. Fan-Out

One node enables many successors.

75. Fan-In

Many predecessors converge into one node/barrier.

76. Fork

Creates parallel branches.

77. Join

Synchronizes branches.

78. Conditional Branch

Only one or several paths activate based on predicate.

79. Exclusive Choice

Exactly one branch.

80. Inclusive Choice

One or more branches.

81. Default Branch

Use only with explicit reason and audit.

82. Branch Selection

Store:

• evaluated context;
• selected path;
• rule version;
• and reason.

83. Branch Drift

A selected branch should not change silently after execution begins.

84. Atomicity Group

A set of nodes with coordinated outcome requirements.

85. All-or-Nothing Group

All required nodes succeed, otherwise compensation/recovery.

86. Best-Effort Group

Independent partial success is allowed.

87. Cutover Group

Coordinates source and target transition.

88. Replacement Group

Example sequence:

prepare target
-> validate target
-> activate target
-> redirect traffic
-> deactivate source

89. Rollback Point

A point before which safe rollback is possible.

90. Point of No Return

After this point, rollback becomes compensation or business remedy.

91. Atomicity Group State

Possible:

• NOT_STARTED;
• PREPARING;
• COMMITTING;
• COMMITTED;
• ROLLING_BACK;
• COMPENSATING;
• FAILED.

92. Distributed Atomicity

Do not pretend multiple systems share one database transaction.

93. Saga

A saga coordinates distributed local transactions and compensations.

94. Orchestration Saga

A central coordinator commands participants.

95. Choreography

Participants react to events without a central workflow owner.

96. Orchestration versus Choreography

97. Hybrid Coordination

Use central orchestration for:

• dependency graph;
• long-running state;
• and business recovery,

while domains own internal execution.

98. Orchestrator Responsibility

The orchestrator should own:

• graph progression;
• dependency state;
• retries policy;
• timeout policy;
• and process-level recovery.

99. Participant Responsibility

Each participant owns:

• local action;
• local consistency;
• local idempotency;
• and outcome evidence.

100. Orchestrator Does Not Own Every Domain Rule

Avoid moving all service/resource semantics into workflow code.

101. Command

An orchestrator sends an explicit instruction.

102. Event

A participant publishes a completed business/technical fact.

103. Command–Event Correlation

Use:

• process ID;
• node ID;
• attempt ID;
• command ID;
• and causation ID.

104. Command Idempotency

The participant must deduplicate repeated commands.

105. Event Idempotency

The orchestrator must deduplicate repeated outcomes.

106. Command Acknowledgement

Transport acceptance differs from business completion.

107. Node Attempt

Each execution attempt has identity.

108. Attempt State

Possible:

• CREATED;
• DISPATCHED;
• ACKNOWLEDGED;
• RUNNING;
• SUCCEEDED;
• FAILED;
• TIMED_OUT;
• UNKNOWN;
• CANCELLED.

109. Attempt Success

Success must include expected outcome evidence.

110. Attempt Failure

Failure should classify:

• retryable;
• permanent;
• unknown;
• business rejection;
• and policy failure.

111. Orchestration Timeout

A timeout means expected response was not observed within a window.

It does not prove remote failure.

112. Unknown Outcome

Use explicit state until reconciled.

113. Retry Boundary

Retry the smallest safe idempotent operation.

114. Node Retry

Creates a new attempt for same logical node.

115. Process Retry

Rare; may restart entire branch or process.

116. Compensation

Runs explicit reverse/remedial action.

117. Compensation Dependency

Compensations may also form a graph.

118. Reverse Topological Compensation

Often compensate completed nodes in reverse dependency order.

119. Compensation Is Not Rollback

External effects may be irreversible or lossy.

120. Compensation Outcome

Possible:

• FULLY_COMPENSATED;
• PARTIALLY_COMPENSATED;
• NOT_COMPENSATABLE;
• and COMPENSATION_FAILED.

121. Manual Recovery Node

Represent human intervention explicitly in graph.

122. Manual Task Assignment

Include:

• owner;
• SLA;
• inputs;
• decision;
• and completion evidence.

123. Human Decision Gate

Possible decisions:

• retry;
• skip;
• choose alternative;
• compensate;
• cancel;
• and accept degraded outcome.

124. Degraded Outcome

May proceed with reduced operational quality only if commercial/product policy permits.

125. Skip Node

Skipping must record:

• reason;
• authority;
• downstream impact;
• and target Product validity.

126. Waive Dependency

Different from skipping work.

It changes prerequisite enforcement.

127. Pause

Pauses graph progression without cancelling.

128. Resume

Re-evaluates dependencies and time validity.

129. Process Suspension

May happen due to:

• customer hold;
• compliance;
• incident;
• or maintenance.

130. Process Cancellation

Stops not-yet-committed work and triggers cancellation/compensation graph.

131. Cancellation Graph

Can differ from forward graph.

132. Cancellation Dependency

Example:

• cancel appointment before releasing workforce;
• terminate supplier order before releasing reservation.

133. Completed Node during Cancellation

Preserve result and determine compensation.

134. In-Flight Node during Cancellation

Send cancellation if supported and reconcile ambiguous outcome.

135. Pending Node during Cancellation

Mark cancelled without dispatch.

136. Irreversible Node

Creates residual outcome requiring business remedy.

137. Replan during Execution

A new graph version may replace pending scope.

138. Handover Boundary

Define which nodes belong to old versus new graph.

139. Active Attempt Protection

Do not dispatch duplicate replacement while old attempt outcome is unknown.

140. Superseded Node

A pending node may be superseded by new Plan.

141. Completed Node Reuse

New Plan may reference existing completion evidence.

142. Graph Diff

Show:

• node additions/removals;
• edge changes;
• barriers;
• schedule;
• and compensation impact.

143. Orchestration Persistence

Persist long-running process state durably.

144. Durable Workflow

State survives:

• process restart;
• deployment;
• node failure;
• and message redelivery.

145. Workflow Timer

Timers must be durable and idempotent.

146. Timer Identity

Store:

• timer ID;
• purpose;
• dueAt;
• process/node;
• and version.

147. Timer Reconciliation

Find missing, duplicate, or overdue timers.

148. Workflow Engine

Can provide:

• durable state;
• timers;
• retries;
• human tasks;
• and history.

149. Workflow Engine Risk

Risks:

• business logic hidden in proprietary DSL;
• history retention cost;
• non-deterministic workflow code;
• and dual ownership.

150. Deterministic Workflow Code

Some engines replay code.

Avoid:

• current time without API;
• random values;
• unversioned external calls;
• and non-deterministic iteration.

151. Workflow Versioning

Running instances may use old workflow definition.

152. Compatible Change

Examples:

• add optional branch after a new marker;
• add new retry policy for future attempts.

153. Incompatible Change

Examples:

• remove state expected by running process;
• reinterpret node identity;
• and change completed milestone meaning.

154. Migration Strategy

Options:

• let old instances finish;
• explicit migration;
• new process version for new Orders;
• or controlled restart from checkpoint.

155. Checkpoint

A stable point from which process can resume/replan.

156. Process Snapshot

Store:

• graph version;
• node states;
• active attempts;
• timers;
• and external references.

157. Event History

Supports:

• replay;
• audit;
• and diagnosis.

158. History Compaction

Long-running processes may need snapshots/compaction.

Preserve critical evidence.

159. State Projection

Provide operator-friendly view.

160. Why Is This Order Waiting?

The system should answer:

Node X is waiting on barrier B.
Barrier B requires A and C.
A completed.
C is held because supplier confirmation is pending.

161. Explainable Orchestration

Expose:

• current frontier;
• blockers;
• selected branch;
• attempts;
• timers;
• and recovery options.

162. Graph Visualization

Useful for:

• design;
• support;
• and incident response.

163. Large Graph

Thousands of nodes require:

• partitioning;
• summaries;
• lazy loading;
• and filtered views.

164. Graph Partitioning

Possible by:

• site;
• region;
• wave;
• fulfillment domain;
• or sub-order.

165. Master Orchestration

Coordinates partition milestones.

166. Local Orchestration

Each partition/domain executes internal graph.

167. Cross-Partition Barrier

Example:

• all regional migrations complete before global cutover.

168. Fan-Out Storm

One node may release thousands of successors.

Use rate limits and staged dispatch.

169. Backpressure

When downstream cannot accept more work, orchestrator reduces dispatch.

170. Queue Depth

Measure per domain/tenant/priority.

171. Rate Limit

Respect downstream API limits.

172. Bulk Command

Can reduce overhead but must preserve per-item identity and outcomes.

173. Partial Bulk Failure

Represent per-item result.

174. Resource Contention

Two graph branches may compete for scarce resource.

175. Lock/Reservation

Use explicit reservation rather than distributed lock where domain-appropriate.

176. Deadlock

Can occur with multiple resource acquisition order.

177. Resource Ordering

Use deterministic acquisition order to reduce deadlock.

178. Reservation Timeout

Release stale holds.

179. Priority Inversion

Low-priority process holds resource needed by high-priority process.

Needs policy, not ad hoc stealing.

180. Starvation

Repeated high-priority work can starve normal work.

181. Fair Scheduling

Consider tenant/customer fairness.

182. Orchestration API

Possible commands:

• StartOrchestration;
• PauseOrchestration;
• ResumeOrchestration;
• CancelOrchestration;
• RetryNode;
• ReconcileNodeOutcome;
• SkipNode;
• WaiveDependency;
• ReplanScope;
• ResolveManualTask.

183. Generic State Update Risk

Do not expose arbitrary node-state mutation.

184. Command Guard

Check:

• process version;
• node state;
• actor authority;
• dependency state;
• and active attempt.

185. Idempotency Key

Required for external effects and operator commands.

186. Orchestration Events

Representative events:

• OrchestrationStarted;
• FulfillmentNodeReady;
• FulfillmentNodeDispatched;
• FulfillmentNodeCompleted;
• FulfillmentNodeFailed;
• BarrierReleased;
• OrchestrationPaused;
• OrchestrationReplanned;
• OrchestrationCompleted.

187. Event Granularity

Avoid publishing every internal heartbeat.

188. Event Payload

Include:

• process;
• graph version;
• node;
• attempt;
• state;
• reason;
• and correlation.

189. Outbox/Inbox

Use outbox for emitted events and inbox for received outcomes.

190. Exactly-Once Myth

End-to-end exactly-once is generally unrealistic across distributed systems.

Design for:

• at-least-once delivery;
• idempotent effects;
• and reconciliation.

191. Observability

Metrics:

• ready node count;
• running node count;
• blocked node count;
• and graph completion.

192. Dependency Metrics

• average predecessors;
• barrier wait time;
• blocked duration;
• and cycle detection failures.

193. Critical-Path Metrics

• planned duration;
• actual duration;
• critical-node delays;
• and slack consumption.

194. Orchestration Metrics

• dispatch latency;
• attempt success;
• retry rate;
• timeout rate;
• and compensation rate.

195. Backpressure Metrics

• queue depth;
• rate-limit hits;
• and dispatch delay.

196. Manual Work Metrics

• manual-task count;
• assignment time;
• decision time;
• and recovery success.

197. Orchestration SLI

Examples:

• zero dependency-violating dispatch;
• all active attempts traceable;
• all terminal processes have terminal required nodes;
• and all unknown outcomes reconciled within target.

Internal targets must be verified.

198. Stuck Process

Examples:

• barrier never releases;
• timer missing;
• active attempt has no heartbeat/outcome;
• manual task unassigned;
• and dependency references superseded node.

199. Stuck Detection

Use:

• state age;
• expected next event;
• timer;
• active attempt;
• and dependency status.

200. Reconciliation

Detect:

• completed process with incomplete required node;
• node running without attempt;
• attempt succeeded but node pending;
• barrier released incorrectly;
• and active node from superseded graph.

201. Recovery Commands

Examples:

• RebuildReadySet;
• ReconcileAttemptOutcome;
• RepairBarrierState;
• ReattachExternalWork;
• ResumeFromCheckpoint;
• ReplanBlockedBranch;
• CorrectGraphMetadata.

202. Incident

Examples:

• dependent dispatched too early;
• duplicate activation;
• cycle introduced during replan;
• compensation executed before unknown outcome resolved;
• and old graph continues after supersession.

203. Incident Containment

Possible:

• pause process;
• stop dispatch;
• freeze active graph;
• reconcile external work;
• protect Inventory/Billing;
• and publish corrected Plan.

204. Graph Smells

• sequence number only;
• no edge type;
• no failure policy;
• and mutable published graph.

205. Orchestration Smells

• orchestration state in memory only;
• retry hidden in client library;
• and message order assumed.

206. Barrier Smells

• polling all predecessors repeatedly;
• no barrier identity;
• and timeout treated as success.

207. Saga Smells

• compensation assumed perfect;
• no point-of-no-return;
• and rollback terminology hides irreversible effects.

208. Workflow Smells

• all domain logic in workflow DSL;
• no workflow version;
• and running instances changed in place.

209. Large-Graph Smells

• loading entire graph for each callback;
• fan-out without rate limit;
• and no partition summary.

210. Anti-Patterns

Sequence Number as Dependency Model

Parallelism and branching disappear.

Message Arrival Order as Business Order

Distributed delivery is unreliable.

Retry Entire Process

Duplicate side effects multiply.

Compensation before Reconciliation

May reverse a successful remote action incorrectly.

Mutable Workflow Definition

Running instances become non-reproducible.

One Central Orchestrator Owns All Domain Logic

Domain boundaries collapse.

Status Polling without Correlation

Outcomes cannot be attributed safely.

211. Dependency Graph Template

## Graph Identity and Version

## Source Product Order / Plan

## Nodes

## Edges

## Barriers

## Milestones

## Atomicity / Cutover Groups

## Branch Conditions

## Scheduling / Critical Path

## Failure Propagation

## Compensation Graph

## Publication / Supersession

## Checksum / Audit

212. Node Template

Node ID:
Type:
Source Order Item:
Domain/owner:
Action:
Expected outcome:
State:
Duration:
Schedule:
Retry policy:
Timeout policy:
Compensation:

213. Edge Template

Edge ID:
From:
To:
Type:
Condition:
Required predecessor outcome:
Failure propagation:
Timeout:
Waiver:
Source rule:

214. Barrier Template

Barrier ID:
Type: AND / OR / N-of-M
Inputs:
Release condition:
Timeout:
Timeout outcome:
Released by:
Released at:

215. Atomicity Group Template

Group ID:
Members:
Policy:
Commit point:
Point of no return:
Compensation order:
Partial outcome:
Owner:

216. Attempt Template

Attempt ID:
Node:
Command ID:
Idempotency key:
External reference:
Dispatched at:
Acknowledged at:
Outcome:
Failure classification:
Reconciliation:

217. Workflow Version Template

Workflow ID/version:
Supported graph versions:
Compatible prior versions:
Migration strategy:
Deterministic constraints:
Timer semantics:
History retention:

218. Orchestration Invariants

Representative invariants:

• node dispatch occurs only when dependencies are satisfied;
• one logical attempt effect is idempotent;
• published graph is immutable;
• cycles are rejected;
• completed nodes are not re-executed without explicit correction;
• unknown outcomes are reconciled before compensation/retry;
• required terminal nodes determine process completion;
• and superseded graph cannot continue dispatching new work.

219. Worked Example: Parallel Installation

After site validation:

• capacity reservation;
• router shipment;
• and appointment booking

run in parallel.

Installation barrier requires all three.

220. Worked Example: Conditional Router Replacement

If current router incompatible:

• replacement branch activates.

Otherwise:

• branch is inactive and activation continues.

Branch selection is recorded.

221. Worked Example: AND Barrier

Service activation waits for:

• access installed;
• router installed;
• configuration approved.

Barrier explains exactly which predecessor remains incomplete.

222. Worked Example: OR Barrier

Service may use:

• primary supplier;
• or approved fallback supplier.

First valid completion releases barrier and cancels/supersedes other branch according to policy.

223. Worked Example: Cutover Group

Target service is prepared and verified.

Traffic switches.

Only after successful verification is old service deactivated.

Point of no return and rollback plan are explicit.

224. Worked Example: Unknown Activation Outcome

Activation command times out.

Node becomes UNKNOWN.

Reconciliation queries activation service.

No retry/compensation occurs until result known.

225. Worked Example: Retry

Supplier API returns transient 503.

New attempt uses same business idempotency key and bounded backoff.

226. Worked Example: Compensation

Router shipped but access installation cancelled.

Compensation requests return logistics.

If physical return impossible, residual cost is recorded.

227. Worked Example: Large Fan-Out

One global approval releases 10,000 site tasks.

Orchestrator dispatches in bounded batches with tenant and downstream rate limits.

228. Worked Example: Replan

Supplier branch fails permanently.

New graph version selects alternate supplier.

Completed site validation and capacity reservation are reused.

229. Worked Example: Workflow Upgrade

Existing Orders continue on Workflow v3.

New Orders use v4.

A controlled migration is available only from a stable checkpoint.

230. Worked Example: Stuck Barrier

Barrier waits on a node that was superseded.

Reconciliation detects invalid edge and corrective replan repairs graph.

231. Senior Engineer Operating Model

Model dependencies as a graph

Not sequence numbers or code order.

Separate orchestration from domain execution

Coordinator versus participant ownership.

Make barriers and milestones first-class

So waiting is explainable.

Treat attempts and outcomes explicitly

Including unknown outcomes.

Version graph and workflow

Running processes need reproducibility.

Reconcile before retry or compensation

Timeout is not proof.

Bound parallelism

Backpressure, fairness, and rate limits.

Preserve completed work through replan

No history rewrite.

Operate graph health

Cycles, stuck barriers, active attempts, and supersession.

232. Internal Verification Checklist

Graph model

• Are nodes and edges first-class?
• Which dependency types exist?
• Are conditions and failure propagation explicit?
• Are cycles detected before publication?

Sequencing

• Is topological readiness computed?
• Can independent work run in parallel?
• Are critical path and slack visible?
• Are calendars/timezones explicit?

Barriers and milestones

• Are AND/OR/quorum barriers supported?
• Which milestones gate customer/product progress?
• Can support explain why a barrier is waiting?
• How are barrier timeout and waiver handled?

Atomicity and compensation

• Which groups are all-or-nothing or best-effort?
• Where is point of no return?
• Is compensation graph explicit?
• How are irreversible outcomes represented?

Orchestrator/participant boundary

• What does the orchestrator own?
• What local idempotency do participants own?
• Are commands and events correlated?
• Is domain logic hidden in workflow definitions?

Retry and unknown outcomes

• Are attempts first-class?
• Does timeout produce UNKNOWN?
• Is reconciliation required before retry?
• How are late callbacks handled?

Workflow durability/versioning

• Is state durable across restart/deployment?
• How are timers stored/reconciled?
• Can old workflow versions continue?
• What migration/checkpoint strategy exists?

Scale and operations

• Are graphs partitioned?
• Is fan-out rate-limited?
• Are backpressure and fairness implemented?
• What reconciliation and recovery commands exist?

233. Practical Exercises

Exercise 1 — DAG design

Create a fulfillment DAG with forks, joins, milestones, and conditional branches.

Exercise 2 — Failure propagation

Define behavior for predecessor failure across five edge types.

Exercise 3 — Cutover group

Model commit point, point of no return, rollback, and compensation.

Exercise 4 — Unknown outcome

Design orchestration behavior after timeout and delayed success.

Exercise 5 — Large fan-out

Schedule 10,000 site tasks with rate limits and fairness.

Exercise 6 — Workflow evolution

Plan a safe migration from one running workflow version to another.

234. Part Completion Checklist

You are done if you can:

• represent fulfillment as a versioned DAG;
• define node, edge, barrier, milestone, and atomicity semantics;
• compute ready sets and topological progression;
• support parallel branches and critical-path analysis;
• separate orchestrator and participant responsibilities;
• model attempts, retries, timeouts, and unknown outcomes;
• define compensation and cancellation graphs;
• version durable workflows;
• partition and throttle large graphs;
• reconcile stuck or inconsistent process state;
• and create an internal orchestration verification backlog.

235. Key Takeaways

1. Fulfillment is a dependency graph, not a sequence number.
2. Topological order permits safe parallelism.
3. Barriers and milestones make waiting explainable.
4. Dependency failure propagation must be explicit.
5. Orchestrators coordinate; domains own local execution.
6. Attempts and unknown outcomes are first-class.
7. Compensation is not a perfect rollback.
8. Published graphs and workflows need versions.
9. Backpressure and fairness matter at enterprise scale.
10. Internal CSG orchestration semantics must be verified.

236. References

Conceptual baseline:

• Directed acyclic graphs, topological ordering, barriers, critical path, and scheduling.
• Saga orchestration, choreography, compensation, durable workflow, and state-machine concepts.
• Distributed systems idempotency, at-least-once delivery, retries, ambiguous outcomes, and reconciliation.
• Telecom/enterprise Product Order, Service Order, Resource Order, and fulfillment coordination practices.

These references do not define internal CSG workflow engines or orchestration implementation.