Build From Scratch: Enterprise Recommendations System

collaborative filtering
matrix factorization
two-tower model
ranking model

closed-loop, governed, observable, low-latency decision platform

user/item/context
candidate generation
ranking
reranking
policy
feedback
training
experimentation
monitoring
governance
operations

product/objective readiness
domain/modeling readiness
API/contract readiness
event/data readiness
candidate generation readiness
eligibility/policy readiness
ranking/reranking readiness
serving/latency readiness
MLOps/model lifecycle readiness
evaluation/experimentation readiness
observability/debug readiness
privacy/safety/security readiness
multi-tenant readiness
cost/capacity readiness
operations/governance readiness
launch readiness

[ ] Target surfaces are defined.
[ ] User/customer problem is clear.
[ ] Recommendation unit is defined.
[ ] Product objective is explicit.
[ ] Primary metrics are defined.
[ ] Guardrail metrics are defined.
[ ] Negative feedback signals are defined.
[ ] Surface-specific objectives are documented.
[ ] Business constraints are documented.
[ ] Long-term value considerations are documented.
[ ] Product owner is accountable for trade-offs.

[ ] User/subject model is defined.
[ ] Item/content/action/document model is defined.
[ ] Context model is defined.
[ ] Surface model is defined.
[ ] Candidate model is defined.
[ ] Slate model is defined.
[ ] Impression/action/event model is defined.
[ ] Item lifecycle/recommendability state exists.
[ ] Identity/session semantics are defined.
[ ] Tenant/domain-specific entities are modeled if needed.

[ ] Recommendation API is schema-first/OpenAPI-defined.
[ ] Request includes request_id, subject, surface, context, limit.
[ ] Response includes slate_id, items, positions, tracking tokens.
[ ] Error/degradation response is defined.
[ ] Debug flag is access-controlled.
[ ] API versioning strategy exists.
[ ] Backward compatibility rules exist.
[ ] Contract tests exist.
[ ] Client integration requirements are documented.
[ ] Raw internal scores are not exposed unintentionally.

[ ] Decision log event exists.
[ ] Impression event exists.
[ ] Viewability event exists if needed.
[ ] Action/click/conversion events exist.
[ ] Negative feedback events exist.
[ ] Event IDs are idempotent.
[ ] event_time and ingestion_time are captured.
[ ] Tracking token joins action to impression.
[ ] Experiment assignment/exposure is logged.
[ ] Event schemas are versioned.
[ ] Event quality dashboards exist.
[ ] Deduplication strategy exists.
[ ] Late/out-of-order handling exists.

[ ] Raw/clean/curated layers exist.
[ ] Data quality gates exist.
[ ] Schema evolution policy exists.
[ ] Event volume anomaly detection exists.
[ ] Bot/internal traffic filtering exists if needed.
[ ] Catalog snapshots exist.
[ ] Identity snapshots exist.
[ ] Policy/eligibility snapshots exist.
[ ] Feature snapshots exist.
[ ] Backfill strategy exists.
[ ] Lineage is tracked.
[ ] Privacy filters are applied.
[ ] Training/evaluation datasets are reproducible.

[ ] Label definitions are explicit.
[ ] Label windows are explicit.
[ ] Label maturity is respected.
[ ] Negative sampling policy is versioned.
[ ] Temporal split is used.
[ ] Leakage checks exist.
[ ] Point-in-time joins are enforced.
[ ] Sample weights are defined.
[ ] Segment labels are monitored.
[ ] Dataset spec is versioned.
[ ] Dataset quality report is produced.

[ ] Candidate source contract exists.
[ ] Candidate provenance is preserved.
[ ] Candidate source quotas are configured.
[ ] Candidate source latency is measured.
[ ] Candidate recall is evaluated.
[ ] Candidate source overlap is measured.
[ ] Cold-start sources exist.
[ ] Anonymous/no-consent sources exist.
[ ] Fallback sources exist.
[ ] Candidate deduplication exists.
[ ] Source contribution funnel exists.
[ ] Candidate pool empty rate is monitored.

[ ] Embedding families are defined.
[ ] Query/item compatibility is enforced.
[ ] Embedding versions are immutable.
[ ] Vector validation gates exist.
[ ] Embedding coverage is monitored.
[ ] ANN index versioning exists.
[ ] ANN recall benchmark exists.
[ ] Index publish is atomic.
[ ] Index rollback exists.
[ ] Delta index/new item path exists if needed.
[ ] Deletion/tombstone path exists.
[ ] Multi-tenant index isolation is safe.
[ ] Vector search latency is monitored.

[ ] Hard eligibility rules are defined.
[ ] Soft ranking features are separated from hard rules.
[ ] Final eligibility validation exists.
[ ] Policy state is versioned.
[ ] Reason codes are logged.
[ ] User suppression is enforced.
[ ] Frequency caps are enforced if needed.
[ ] Inventory/availability checks exist if applicable.
[ ] Permission checks exist for enterprise.
[ ] Tombstone/denylist exists.
[ ] Fail-open/fail-closed policy is documented.

[ ] Feature registry exists.
[ ] Feature ownership is defined.
[ ] Feature types and defaults are defined.
[ ] Freshness SLA exists per feature group.
[ ] Offline-online parity is tested.
[ ] Missing/stale feature metrics exist.
[ ] Feature access/privacy metadata exists.
[ ] Feature versioning exists.
[ ] Backfill and deprecation processes exist.
[ ] Feature cost is tracked.
[ ] High-importance features are monitored tightly.

[ ] Long-term profile is separated from session state.
[ ] Session state TTL exists.
[ ] Suppression state is fresh.
[ ] Recent seen/exposure state exists.
[ ] Identity resolution is handled.
[ ] Anonymous-to-logged transition is handled.
[ ] Profile reset/delete is supported.
[ ] Consent changes affect profile use.
[ ] Profile freshness/coverage is monitored.
[ ] Hot key strategy exists.

[ ] Ranking objective is documented.
[ ] Model type is appropriate for maturity.
[ ] Baseline ranker exists.
[ ] Feature set is available online.
[ ] Model output semantics are clear.
[ ] Calibration is evaluated if probabilities used.
[ ] Score composition is governed.
[ ] Latency budget is met.
[ ] Model fallback exists.
[ ] Model version is logged.
[ ] Segment offline metrics pass.
[ ] Guardrail metrics pass.

[ ] Slate policy is versioned.
[ ] Diversity rules exist if needed.
[ ] Novelty/freshness controls exist if needed.
[ ] Frequency/fatigue controls exist.
[ ] Duplicate/dedup group rules exist.
[ ] Business/sponsored caps exist.
[ ] Exploration slots are safety-gated.
[ ] Relevance floor exists.
[ ] Final hard validation occurs after rerank.
[ ] Slate diagnostics are logged.
[ ] Slate metrics are monitored.

[ ] Batch use cases are justified.
[ ] Subject selection respects consent.
[ ] Precomputed list carries lineage.
[ ] generated_at/expires_at are stored.
[ ] TTL is defined by surface.
[ ] More items are stored than final visible count.
[ ] Final online eligibility check exists.
[ ] Invalidation/tombstone strategy exists.
[ ] Batch validation gates exist.
[ ] Atomic publish/rollback exists.
[ ] Store hit/stale/rejection metrics exist.

[ ] Latency budget is defined by stage.
[ ] Deadline propagation exists.
[ ] Downstream timeouts exist.
[ ] Parallel candidate generation is controlled.
[ ] Batch feature/model calls exist.
[ ] Fallback hierarchy exists.
[ ] Circuit breakers exist where needed.
[ ] Bulkheads isolate dependencies.
[ ] Load shedding modes exist.
[ ] Cache strategy is defined.
[ ] Tracking tokens generated per response.
[ ] Decision log emitted asynchronously.
[ ] p95/p99 monitored.

[ ] Cacheable data is classified by safety risk.
[ ] Cache keys include tenant/privacy/context/version.
[ ] TTL policy exists per data type.
[ ] Local/distributed cache roles are clear.
[ ] Cache stampede mitigation exists.
[ ] Stale-while-revalidate used only where safe.
[ ] Final validation protects stale cached lists.
[ ] Cache outage behavior is tested.
[ ] Cache hit/miss/stale metrics exist.
[ ] Privacy/experiment contamination tests exist.

[ ] Failure taxonomy is documented.
[ ] Fail-open/fail-closed matrix exists.
[ ] Safe fallback hierarchy exists.
[ ] Safe empty response is acceptable where needed.
[ ] Optional source failure degrades gracefully.
[ ] Critical permission/policy failures fail closed.
[ ] Model fallback exists.
[ ] Event logging failure buffers/alerts.
[ ] Runbooks exist.
[ ] Resilience tests/game days exist.
[ ] Kill switches exist.

[ ] Model versions are immutable.
[ ] Model bundle includes artifact, schema, feature set, calibration, runtime.
[ ] Training dataset version is recorded.
[ ] Label versions are recorded.
[ ] Code/container version is recorded.
[ ] Offline/segment metrics are stored.
[ ] Compatibility gates exist.
[ ] Approval workflow exists.
[ ] Shadow/canary lifecycle exists.
[ ] Production route pointer exists.
[ ] Rollback bundle exists.
[ ] Model owner is defined.

[ ] Training is orchestrated, not manual.
[ ] Dataset spec is versioned.
[ ] Feature/label snapshots are pinned.
[ ] Negative sampling config is recorded.
[ ] Random seeds are recorded.
[ ] Environment/container is pinned.
[ ] Experiment tracking exists.
[ ] Validation gates exist.
[ ] Artifacts are checksummed.
[ ] Training and promotion are separate.
[ ] Rerun/backfill policy exists.
[ ] Privacy/tenant scope is explicit.

[ ] Retrieval recall metrics exist.
[ ] Ranking metrics exist.
[ ] Calibration metrics exist if needed.
[ ] Slate-level metrics exist.
[ ] Negative feedback metrics exist.
[ ] Segment metrics are mandatory.
[ ] Cold-start metrics exist.
[ ] Offline limitations are documented.
[ ] Champion-vs-challenger report exists.
[ ] Evaluation code is versioned.
[ ] Offline metrics are compared with online outcomes over time.

[ ] Experiment registry exists.
[ ] Hypothesis is required.
[ ] Primary metric is pre-defined.
[ ] Guardrails are pre-defined.
[ ] Randomization unit is appropriate.
[ ] Deterministic assignment exists.
[ ] Exposure logging exists.
[ ] Treatment-applied logging exists.
[ ] SRM check exists.
[ ] Cache variant isolation exists.
[ ] Ramp/rollback plan exists.
[ ] Segment analysis exists.
[ ] Delayed metrics are handled.

[ ] Request tracing exists.
[ ] Stage latency metrics exist.
[ ] Candidate diagnostics exist.
[ ] Filter reasons are logged.
[ ] Feature missing/freshness metrics exist.
[ ] Model score distributions are monitored.
[ ] Slate metrics exist.
[ ] Fallback tier/reason is logged.
[ ] Event volume/lag/join metrics exist.
[ ] Decision log completeness is monitored.
[ ] Segment dashboards exist.
[ ] Version tags exist everywhere.
[ ] Alerts are actionable and owned.

[ ] Request/slate IDs are available.
[ ] Debug trace explains candidate source, filter, feature, score, rule.
[ ] Debug access is restricted.
[ ] Sensitive data is redacted.
[ ] Replay can reconstruct decision or is planned.
[ ] Random seeds/model/policy versions are logged.
[ ] Bad recommendation playbook exists.
[ ] Support escalation path exists.
[ ] Regression tests are added after incidents.

[ ] Privacy modes are defined.
[ ] Consent is resolved before personalization.
[ ] Consent unknown fails safe.
[ ] Candidate sources are privacy-gated.
[ ] Feature sets are privacy-gated.
[ ] Non-personalized path exists.
[ ] Deletion workflow exists.
[ ] Retention policy exists.
[ ] Training privacy filters exist.
[ ] Debug traces are redacted.
[ ] LLM context is minimized.
[ ] Privacy regression tests exist.

[ ] Policy taxonomy exists.
[ ] Recommendability by surface exists.
[ ] Candidate sources filter policy-ineligible items.
[ ] Final safety validation exists.
[ ] Tombstone/denylist exists.
[ ] Trust/risk signals exist.
[ ] Trending/popularity is abuse-resistant.
[ ] Exploration is safety-gated.
[ ] Sponsored cannot override safety.
[ ] Safety guardrails exist in experiments.
[ ] Kill switches exist.
[ ] Safety incident runbook exists.

[ ] All callers are authenticated.
[ ] Subject authorization exists.
[ ] Tenant isolation exists.
[ ] Feature access control exists.
[ ] Debug RBAC/audit exists.
[ ] Model artifacts are integrity-checked.
[ ] Vector indexes enforce tenant/security scope.
[ ] Admin changes are approved/audited.
[ ] Secrets are managed securely.
[ ] Security alerts exist.
[ ] LLM tools are least-privilege.
[ ] Security regression tests exist.

[ ] Tenant config schema exists.
[ ] Effective config resolver exists.
[ ] Tenant ID is propagated.
[ ] Tenant in cache/profile/feature/index/log keys.
[ ] Tenant-specific rules/fallback/features supported.
[ ] Global mandatory safety cannot be disabled.
[ ] Config validation exists.
[ ] Config diff and approval exist.
[ ] Tenant dashboards exist.
[ ] Tenant onboarding/offboarding exist.
[ ] Config drift is monitored.

[ ] QPS tracked by surface/tenant.
[ ] Candidate scores/sec calculated.
[ ] Feature values/sec estimated.
[ ] Vector query capacity planned.
[ ] Model inference cost tracked.
[ ] Batch scoring cost tracked.
[ ] Cache economics monitored.
[ ] LLM cost/quota exists if used.
[ ] Cost attributed by service/surface/tenant.
[ ] Load tests run.
[ ] Degradation modes exist.
[ ] Capacity includes rollout/failover headroom.

[ ] Service owners defined.
[ ] Artifact owners defined.
[ ] On-call exists.
[ ] Runbooks exist.
[ ] Incident severity policy exists.
[ ] Postmortem process exists.
[ ] Launch review process exists.
[ ] Model/feature/rule/experiment review exists.
[ ] Dashboard/alert owners exist.
[ ] Release timeline exists.
[ ] Deprecation process exists.
[ ] Cost review cadence exists.

[ ] API contract approved.
[ ] Event tracking validated.
[ ] Inactive/policy-blocked item exclusion tested.
[ ] Privacy mode tested.
[ ] Fallback tested.
[ ] Decision log verified.
[ ] Impression/action join verified.
[ ] Dashboard live.
[ ] Alerts configured.
[ ] Load test passed.
[ ] Rollback tested.
[ ] On-call notified.
[ ] Experiment plan approved.
[ ] Safety/privacy/security review passed.
[ ] Product owner accepts trade-offs.

internal traffic
shadow
1% canary
5%
10%
25%
50% A/B
100% rollout if successful

code
model
index
feature set
rule bundle
slate policy
candidate source config
experiment
tenant config
batch list
LLM prompt/config

manual recommendations
no event loop
no monitoring

API
basic candidates
eligibility
heuristic ranker
events
fallback
dashboard

training dataset
offline eval
GBDT ranker
A/B testing
model registry
feature monitoring

multi-source retrieval
embeddings/ANN
feature store
model lifecycle
batch scoring
observability/debug/replay

privacy/safety/security
multi-tenant config
cost attribution
operating model
policy governance

bandits/OPE
causal long-term value
multimodal/deep ranking
LLM augmentation
fairness/exposure optimization

Which model should we use?

What decision are we making?
What data proves it?
What can go wrong?
How do we know it went wrong?
How do we roll back?
How do we respect privacy?
How do we prevent abuse?
How do we measure long-term value?
Who owns it?
What is the cost?

Production-grade ecommerce recommendation skeleton

Design YouTube/TikTok feed recommendations.
Design Amazon PDP similar products.
Design LinkedIn job recommendations.
Design enterprise next-best-action system.
Design marketplace seller-aware recommendations.
Design cold-start recommendation for new items.
Design RecSys experimentation platform.
Design feature store for ranking.
Design vector retrieval pipeline.
Design bad recommendation debugging system.

001-006: mental model and architecture
007-017: events, feedback, and data foundation
018-023: baseline recommenders
024-032: candidate generation/retrieval
033-042: ranking
043-050: reranking, policy, exploration, causal, LLM
051-062: production platform architecture
063-068: evaluation, experimentation, observability
069-074: governance, safety, security, enterprise constraints
075-080: implementation tracks and close

No recommendation without eligibility.
No personalization without consent.
No model without lineage.
No launch without guardrails.
No experiment without exposure logging.
No cache without correct keys.
No debug without access control.
No feature without owner.
No metric without definition.
No production without rollback.

1. What decision are we making?
2. What is the recommendation unit?
3. What is the objective?
4. What are guardrails?
5. What candidates can enter?
6. What hard filters apply?
7. What features are used?
8. What model/ranker is active?
9. What reranking rules apply?
10. How is feedback logged?
11. How is training data built?
12. How do we evaluate offline?
13. How do we test online?
14. How do we debug bad recommendations?
15. How do we detect quality drift?
16. How do we respect privacy/consent?
17. How do we enforce safety/security?
18. How do we roll back?
19. Who owns each artifact?
20. What does this cost at scale?

minimum skeleton
-> baseline recommender
-> feedback loop
-> offline evaluation
-> GBDT ranker
-> A/B testing simulation
-> observability/debugging
-> privacy/safety/security gates
-> production readiness review