This part does not assume CSG's internal incident process, severity model,
SLO/SLA definitions, on-call rotation, customer communication process,
rollback mechanism, observability stack, ticketing system, or RCA template.

All operational process details must be verified internally.

Production ownership is not only writing correct code.
Production ownership means the team can detect, triage, mitigate, explain,
repair, and prevent failures under real customer pressure.

How do we know the service is unhealthy?
Who gets paged?
What is the customer impact?
What is the fastest safe mitigation?
What evidence do we have?
What changed recently?
What should we roll back, disable, shed, or degrade?
How do we prevent recurrence without blaming individuals?

1. Software lifecycle
   code -> review -> build -> test -> deploy -> operate -> change again

2. Incident lifecycle
   signal -> triage -> declare -> mitigate -> recover -> analyze -> improve

"We will figure it out when it happens."

"We know the likely failure modes, we know what signals detect them,
we know how to mitigate them, and we continuously improve after incidents."

[ ] Source code ownership
[ ] Runtime ownership
[ ] Alert ownership
[ ] Dashboard ownership
[ ] Runbook ownership
[ ] Deployment ownership
[ ] Data ownership
[ ] Dependency ownership
[ ] Customer impact ownership
[ ] Post-incident action ownership

Who owns it?
How do we escalate?
What evidence do they need?
What is the dependency contract?
What is our mitigation while waiting?

SLI: Service Level Indicator
A measurement of service behavior.
Example: successful request ratio, p95 latency, message processing lag.

SLO: Service Level Objective
Target level for an SLI over a time window.
Example: 99.9% of quote creation requests succeed over 30 days.

SLA: Service Level Agreement
External/customer/legal commitment, often with business consequences.

Error budget:
The allowed unreliability within an SLO window.

SLO: 99.9% successful quote submission over 30 days.
Error budget: 0.1% failed quote submissions.

An SLA is not the same as an SLO.
An SLO should guide engineering decisions.
An SLA may guide customer/legal obligations.

CPU > 80%

Quote creation p95 latency > SLO threshold for 10 minutes
and successful request ratio below target.

What is broken?
Who is affected?
How urgent is it?
What is the first runbook?
What dashboard should I open?
What changed recently?

Start with customer-facing symptoms.
Use cause alerts to explain and mitigate.
Do not page humans for non-actionable noise.

customer impact
revenue/business impact
data correctness risk
security/compliance risk
duration
blast radius
availability of workaround
regulatory/reporting obligations

API returns 200 but writes incorrect pricing data.
This may be more severe than a clean 500 failure.

Mitigation is not root cause fix.
Recovery is not prevention.
RCA is not complete until actions are owned and tracked.

[ ] Confirm signal is real.
[ ] Identify affected service/tenant/region/environment.
[ ] Check customer-facing symptoms first.
[ ] Declare incident if threshold is met.
[ ] Open incident channel/bridge if internal process requires it.
[ ] Assign incident commander and scribe.
[ ] Freeze unrelated deploys if needed.
[ ] Identify recent changes.
[ ] Open primary dashboard and logs.
[ ] Decide first mitigation path.
[ ] Communicate initial status.

[ ] Everyone debugging silently.
[ ] Multiple people applying fixes without coordination.
[ ] Jumping to root cause before confirming impact.
[ ] Restarting everything without evidence.
[ ] Hiding uncertainty.

We are seeing elevated 5xx and p95 latency on quote submission in production
starting at 10:42 UTC. Impact appears limited to tenant group A based on current
dashboards. We are checking recent deployment and DB connection pool saturation.
Next update in the incident channel after mitigation decision.

Which endpoint?
Which status codes?
Which exception classes?
Which tenants?
Was there a deployment/config change?
Is DB/cache/downstream dependency failing?

Is latency server-side, gateway-side, or client-side?
Is thread pool saturated?
Is DB query slow?
Is connection pool saturated?
Is downstream call timing out?
Is CPU throttled?

Which consumer group?
Which partitions?
Did consumer crash or slow down?
Is a poison message blocking progress?
Is downstream dependency slow?
Was partition count or assignment changed?

Which query/table?
Which transaction is blocking?
Was a migration deployed?
Is a batch job running?
Is isolation level changed?

Was memory limit changed?
Did traffic increase?
Is there large payload or buffering?
Did GC behavior change?
Is native memory/thread count high?

1. Purpose
2. Symptoms covered
3. Severity criteria
4. Dashboards
5. Logs and queries
6. First checks
7. Mitigation options
8. Rollback options
9. Escalation path
10. Validation steps
11. Customer impact notes
12. Known false positives
13. Post-incident follow-up

# Runbook: Quote Submission 5xx Spike

## Symptoms
- 5xx ratio above threshold for quote submission endpoint
- error budget burn alert

## First checks
- Dashboard: API availability by endpoint
- Logs: error_code, exception_class, tenant, correlation_id
- Deployment: last release/config change
- Dependencies: DB pool, Redis, Kafka, downstream pricing service

## Mitigation
- Disable feature flag <flag-name> if new path is involved
- Roll back deployment <deployment-name> if release correlated
- Shed low-priority traffic if saturation is confirmed
- Escalate DB/platform if dependency-level outage

## Validation
- 5xx returns to baseline
- p95 latency returns to target
- customer support confirms no new reports
- reconciliation job shows no data drift

DB schema was contracted.
Event schema changed incompatibly.
New version wrote data old version cannot read.
Feature flag only controls read path, not write side effect.
External dependency contract changed.

Every production change should have a mitigation plan before deployment.
Rollback is one mitigation option, not the only strategy.

Who was affected?
Which tenants/customers/regions?
Which business operations failed or degraded?
How many requests failed?
Were any successful responses actually incorrect?
Was data lost, duplicated, delayed, or corrupted?
Was there security or privacy exposure?
Is reconciliation required?
Do customers need notification or support action?

A clean failure is often easier to recover than a silent incorrect success.

[ ] when first bad signal appeared
[ ] when alert fired
[ ] when human acknowledged
[ ] when incident was declared
[ ] recent deployments/config changes
[ ] mitigation actions and exact times
[ ] customer impact windows
[ ] recovery time
[ ] validation time
[ ] communication updates

10:42 UTC - 5xx begins on POST /quotes
10:45 UTC - burn-rate alert fires
10:47 UTC - on-call acknowledges
10:50 UTC - incident declared SEV2
10:55 UTC - recent deployment identified
11:02 UTC - feature flag disabled
11:07 UTC - 5xx returns to baseline
11:20 UTC - reconciliation confirms no duplicate quotes
11:45 UTC - RCA draft started

Use absolute times and timezone.
Do not rely on vague phrases like "earlier" or "a bit later".

1. What happened?
2. What was the customer impact?
3. How was it detected?
4. Why did it happen?
5. Why was it not prevented?
6. Why was it not detected earlier?
7. What mitigated it?
8. What corrective actions will reduce recurrence?
9. What preventive actions will reduce blast radius?
10. What follow-up items have owners and due dates?

Developer made a mistake.

The API accepted a backward-incompatible enum value because contract tests did not
cover older generated clients. The release pipeline did not run compatibility
diff against the published OpenAPI contract. The canary did not include traffic
from the affected client type.

Bad: Improve testing.
Good: Add contract test that verifies old v1 client can parse new quote response.

Bad: Add monitoring.
Good: Add burn-rate alert for quote submission 5xx with dashboard link and runbook.

owner
priority
due date
acceptance criteria
tracking issue/link

How will we know this change is unhealthy?
Can we disable it?
Can we roll it back?
Can old and new versions coexist?
Can DB schema support both versions?
Can event consumers handle the change?
Can support/customer teams understand the impact?
Is the runbook updated?

No observability plan -> not production-ready.
No rollback/mitigation plan -> risky deployment.
No compatibility check -> possible consumer incident.
No data correction plan -> dangerous for quote/order flows.

gateway -> ingress -> service -> pod -> JAX-RS filter -> resource method -> service layer -> DB/Kafka/downstream

trace_id
correlation_id
causation_id
request_id
tenant when allowed
operation name

raw payload
PII
secret
full token
high-cardinality metric labels

POST /quotes returns 500.

validation mapper bug
DB lock timeout
Redis timeout
Kafka publish failure
pricing service timeout
feature flag config mismatch
JVM OOMKilled
wrong tenant routing

customer-facing SLI recovers
error budget burn stops
dependency pressure falls
backlog is understood
no hidden data correctness issue remains

current status
known impact
unknowns
mitigation underway
next update expectation
owner/channel

speculation presented as fact
blame
technical overload for non-technical audience
silence during uncertainty
promising root cause too early

We have confirmed elevated 5xx on quote submission for production starting 10:42 UTC.
Current evidence points to DB connection pool saturation after the latest deployment,
but root cause is not final. We disabled the new pricing enrichment flag at 11:02 UTC.
5xx is decreasing. Next update after validation and reconciliation check.

We are investigating degraded quote submission affecting a subset of customers.
A mitigation has been applied and we are validating recovery. We will provide
an update when validation is complete.

Service ownership:
[ ] Owner team is documented.
[ ] On-call/escalation path is documented.
[ ] Dependencies and owners are documented.

SLO/SLA:
[ ] Customer-facing SLIs are defined.
[ ] SLO targets and windows are defined.
[ ] Error budget policy is known.
[ ] SLA obligations are understood if applicable.

Alerting:
[ ] Symptom alerts exist.
[ ] Cause alerts support triage.
[ ] Alerts have dashboard links.
[ ] Alerts have runbook links.
[ ] Noise/non-actionable alerts are controlled.

Runbooks:
[ ] Runbooks exist for top failure modes.
[ ] Runbooks are tested or reviewed.
[ ] Rollback/kill-switch procedures are documented.
[ ] Data reconciliation procedures are documented.

Incident response:
[ ] Severity model is defined.
[ ] Incident roles are known.
[ ] Timeline process exists.
[ ] RCA/postmortem process exists.
[ ] Action items are tracked.

Customer impact:
[ ] Impact assessment template exists.
[ ] Tenant/customer blast radius can be queried.
[ ] Correctness incidents trigger reconciliation.
[ ] Communication path to support/customer teams exists.

Incident process:
[ ] What severity levels are used internally?
[ ] Who declares incidents?
[ ] Which channel/tool is used?
[ ] Who communicates customer impact?
[ ] What is the RCA/postmortem template?

Operational ownership:
[ ] Which team owns Quote & Order runtime?
[ ] Which team owns database/platform/network dependencies?
[ ] Which team owns Kafka/RabbitMQ/Redis dependencies?
[ ] Which team owns cloud/Kubernetes/GitOps pipeline?

SLO/SLA:
[ ] Are there formal SLOs?
[ ] Are there external SLAs?
[ ] What are the critical business operations?
[ ] Which endpoints/jobs/events map to those operations?

Dashboards and alerts:
[ ] Where are service dashboards?
[ ] Which alerts page the team?
[ ] Are alerts tied to runbooks?
[ ] Are burn-rate alerts used?
[ ] Are deployment events visible in dashboards?

Mitigation:
[ ] How is rollback done?
[ ] Are feature flags/kill switches available?
[ ] Can Kafka consumers/jobs be paused safely?
[ ] Can traffic be shed safely?
[ ] Is reconciliation available after partial failure?

Data correctness:
[ ] How are duplicate quotes/orders detected?
[ ] How are stuck workflows detected?
[ ] How is pricing/catalog correctness validated?
[ ] Which reconciliation jobs exist?
[ ] Who approves manual correction?

[ ] What customer-facing behavior changes?
[ ] What telemetry proves it works?
[ ] What telemetry proves it is failing?
[ ] What alert/runbook needs update?
[ ] What is the rollback or mitigation plan?
[ ] Does DB/event/API compatibility hold during rollout?
[ ] Can old and new versions run together?
[ ] Is there a feature flag or kill switch for risky behavior?
[ ] Is there a reconciliation plan if partial failure occurs?
[ ] Is customer impact measurable?

[ ] It changes production behavior without observable signal.
[ ] It changes DB schema without expand-contract safety.
[ ] It changes event contract without compatibility proof.
[ ] It introduces retry without retry budget.
[ ] It adds scheduled job without idempotency/locking/monitoring.
[ ] It changes auth/tenant logic without negative tests and audit trail.
[ ] It depends on manual rollback that is not documented.

[ ] API 5xx spike after deployment
[ ] DB connection pool exhaustion
[ ] PostgreSQL deadlock storm
[ ] Kafka consumer lag with poison message
[ ] Redis unavailable
[ ] cloud secret rotation failure
[ ] DNS private endpoint misresolution
[ ] TLS certificate expiry
[ ] bad feature flag configuration
[ ] reconciliation finds duplicate orders

detection
triage
communication
mitigation
rollback/flag disable
dashboard usefulness
runbook accuracy
post-incident action tracking

Good code reduces defects.
Good tests reduce escaped defects.
Good observability reduces diagnosis time.
Good runbooks reduce chaos.
Good rollout strategy reduces blast radius.
Good incident response reduces customer harm.
Good RCA reduces recurrence.

I implemented the feature.

I implemented the feature, made it observable, made it safe to roll out,
made it safe to disable, documented how it fails, and ensured the team can
recover when reality disagrees with our assumptions.