Enterprise Recommendation Platform

Apakah teknologinya keren?

Apakah sistem ini menyelesaikan problem dengan benar?
Apakah contract jelas?
Apakah data loop sehat?
Apakah failure mode aman?
Apakah bisa di-debug?
Apakah bisa di-scale?
Apakah bisa dipantau?
Apakah governance cukup?
Apakah tim bisa mengoperasikannya?

serve recommendations for multiple surfaces
support personalized/contextual/non-personalized modes
support multiple candidate sources
rank and rerank candidates
enforce eligibility, safety, policy, privacy, permission
log decisions/impressions/actions
support offline training/evaluation
support online experiments
support batch/precomputed recommendations
support debug and replay
support tenant-specific configuration

low latency
high availability
safe degradation
observability
security
privacy
auditability
cost control
model lifecycle governance
operational ownership

home_feed:
  availability: 99.9%
  p95_latency_ms: 200
  p99_latency_ms: 500
  empty_slate_rate: <0.1%
  decision_log_success: >99.9%
  critical_policy_violation: 0

enterprise_next_action:
  availability: 99.95%
  p95_latency_ms: 300
  permission_violation: 0
  audit_log_success: >99.99%
  safe_empty_allowed: true

Recommendation Serving
Candidate Generation
Eligibility and Policy
Ranking
Slate Construction
Feature/Profile
Event and Feedback
Training and Evaluation
Model/Index Registry
Experimentation
Configuration
Observability and Debugging
Governance/Security/Privacy

Rec API orchestrates deadline, fallback, trace, and decision integrity.

request_id
subject
surface
context
privacy mode
tenant if applicable
limit
debug flag

Can subject be spoofed?
Is tenant required?
Is privacy mode resolved server-side?
Is surface valid?
Can caller request debug?
Are context dimensions complete enough for eligibility?

request_id
slate_id
items
positions
tracking tokens
reason codes
metadata

model version
policy version
fallback flag
experiment variants

Which sources exist?
What is each source’s responsibility?
What are source quotas?
What is source latency?
What is source recall?
What source produced each item?
How are candidates deduped?
How are stale/invalid candidates filtered?

home_feed:
  personalized_two_tower:
    quota: 800
    criticality: preferred
  recent_item_similar:
    quota: 300
    criticality: preferred
  trending_safe:
    quota: 200
    criticality: fallback
  editorial_safe:
    quota: 50
    criticality: fallback

embedding family defined?
query/item embedding compatible?
index versioned?
ANN recall benchmark exists?
delta index exists for freshness?
deletion/tombstone path exists?
multi-tenant filter safe?
index publish atomic?
rollback possible?

item/action active
recommendable on surface
region/locale/age constraints
tenant/permission
user suppression
frequency cap
inventory/availability
safety policy
business campaign validity

Which checks are hard?
Which checks are soft?
Which fail closed?
Which can use stale cache?
Are reason codes logged?
Is final validation present?

feature definitions versioned?
feature ownership exists?
freshness SLA defined?
offline/online parity validated?
privacy metadata present?
missing/default policy defined?
feature cost known?
feature monitoring exists?

long-term vs session state separated?
suppression state fresh?
identity merge/split handled?
consent changes handled?
profile reset/delete handled?
hot keys handled?
profile debug view safe?

ranking objective explicit?
model type appropriate?
features available online?
calibration needed?
score composition governed?
latency within budget?
fallback ranker exists?
model version logged?
segment evaluation passed?

diversity constraints?
frequency/fatigue?
business rules?
sponsored caps/disclosure?
fairness/exposure?
exploration slots?
relevance floor?
layout constraints?
final hard checks?

decision logs
impressions
viewability if needed
actions/clicks
conversions
negative feedback
deletion/suppression
catalog/policy changes

Are events idempotent?
Are schemas versioned?
Are event_time and ingestion_time tracked?
Can actions join to impressions?
Are tracking tokens reliable?
Is event lag monitored?

dataset spec versioned?
temporal split?
point-in-time joins?
label windows mature?
negative sampling policy?
privacy filters?
tenant scope?
data quality gates?
lineage?

model registry exists?
model bundle includes feature set/calibration/runtime?
offline metrics stored?
segment metrics stored?
shadow/canary/production lifecycle?
rollback bundle?
owner?
approval?
monitoring?

which surfaces use batch?
subject selection consent-aware?
list TTL?
final online eligibility?
store more than visible count?
lineage?
batch validation?
rollback?
fallback?

experiment registry?
deterministic assignment?
randomization unit?
exposure logging?
treatment-applied logging?
cache variant isolation?
primary metric?
guardrails?
SRM check?
segment analysis?
rollback?

retrieval recall?
ranking NDCG/MRR/AUC/logloss?
calibration?
slate diversity/novelty?
negative feedback?
segments?
cold-start?
counterfactual limitations documented?

online serving
candidate generation
feature freshness/missing
ranking score distribution
slate quality
feedback events
experiments
model drift
embedding/index health
business metrics
tenant health

Can we debug a bad recommendation?
Can we detect silent quality regression?
Can we attribute change to model/config/source?

why this item?
why this position?
why not filtered?
which source?
which features?
which model?
which rules?
which experiment?
which fallback?

privacy modes defined?
consent checked before personalization?
non-personalized path real?
feature privacy metadata?
training privacy filters?
deletion workflow?
retention?
debug redaction?
LLM context minimization?

policy taxonomy?
recommendability by surface?
candidate generation filters?
final validation?
tombstone/denylist?
trust signals?
abuse-resistant trending?
safety guardrails in experiments?
kill switches?
incident runbook?

service auth?
subject authorization?
tenant isolation?
feature access control?
debug RBAC/audit?
model artifact integrity?
vector index access?
admin approval?
secret management?
LLM least privilege?

effective config resolver?
tenant ID propagated?
tenant-specific config versioned?
tenant-specific model/rules/features/fallback?
global mandatory safety non-overridable?
tenant dashboards?
tenant onboarding/offboarding?
config audit?

QPS by surface?
candidate scores/sec?
feature values/sec?
vector queries/sec?
model inference cost?
cache hit/stale metrics?
batch scoring volume?
LLM cost?
tenant cost?
degradation modes?

dependency timeouts?
circuit breakers?
bulkheads?
fallback hierarchy?
fail-open vs fail-closed matrix?
safe empty?
event logging degradation?
model fallback?
cache outage plan?
runbooks?

artifact owners?
model review?
experiment review?
feature review?
rule/config review?
tenant config review?
on-call?
incident process?
objective governance?
cost ownership?
deprecation process?

ADR-001: modular monolith vs microservices
ADR-002: recommendation unit by domain
ADR-003: event schema standard
ADR-004: candidate source contract
ADR-005: ranking model deployment strategy
ADR-006: feature store architecture
ADR-007: privacy mode enforcement
ADR-008: tenant isolation strategy
ADR-009: experiment assignment unit
ADR-010: fallback/degradation policy

Start with modular monolith for Rec API, candidate, ranking, reranking.
Split model runtime and event ingestion separately.

Kafka for decision/impression/action events.
Raw -> clean -> curated data layers.
Training dataset builder uses curated events and feature snapshots.

Candidate generation portfolio: two-tower, item-to-item, content, trending, editorial.

Use GBDT ranker first, then deep model after data/feature platform matures.

Support three privacy modes: personalized, contextual_only, non_personalized.

Shared infrastructure with logical tenant isolation for most tenants.
Dedicated vector index for high-risk/large tenants.

Architecture deck starts with neural model and ignores events, logging, safety, fallback.

Candidate sources are assumed clean.

Metrics show CTR drop but no model/policy/source version dimension.

All surfaces use same ranking objective.

cache key = surface:user

Model launched because NDCG improved.

Every artifact exists, no one owns it.

[ ] Requirements and non-goals are explicit.
[ ] Surfaces and objectives are defined.
[ ] Recommendation units are defined.
[ ] Request/response contracts are versioned.
[ ] Event contracts are versioned.
[ ] Candidate source contract exists.
[ ] Candidate provenance is logged.
[ ] Eligibility/policy/final validation exist.
[ ] Feature registry/store design exists.
[ ] Profile/session/suppression design exists.
[ ] Ranking contract and model route exist.
[ ] Reranking/slate policy exists.
[ ] Tracking tokens and decision logs exist.
[ ] Offline training dataset builder exists.
[ ] Model/index registries exist.
[ ] Evaluation gates exist.
[ ] Experiment infrastructure exists.
[ ] Observability dashboards/alerts exist.
[ ] Debug/replay capability exists.
[ ] Privacy modes and consent enforcement exist.
[ ] Safety/policy enforcement exists.
[ ] Security/tenant isolation exists.
[ ] Cost/capacity model exists.
[ ] Fault tolerance/degradation plan exists.
[ ] Operating ownership/governance exists.

What happens if the ranker times out?
What happens if consent is unknown?
What happens if item is deleted after index build?
How do we explain why item was shown?
How do we know model v2 is better?
How do we roll back model/index/config?
How do we prevent tenant data leakage?
How do we support no-personalization users?
How do we know feature X is stale?
How do we attribute click to recommendation?
How do we detect source degradation?
How do we keep cost bounded?
How do we handle bad recommendation report?
How do we prevent sponsored from overriding safety?
How do we onboard a new tenant?

API
candidate sources
eligibility
heuristic ranker
slate policy
events
decision logs
dashboard
fallback
privacy/safety basics

clean events
profile updates
training dataset builder
offline evaluation
GBDT ranker
model registry
A/B testing

two-tower retrieval
embeddings
ANN index versioning
feature store
batch scoring
model lifecycle
drift monitoring

multi-tenant config
security hardening
privacy deletion
safety classifiers
cost attribution
operating model

bandits
causal/OPE
deep/multimodal models
LLM augmentation
fairness/exposure optimization

data -> candidates -> ranking -> slate -> exposure -> feedback -> learning -> deployment -> monitoring -> governance

Every decision must be explainable, observable, versioned, safe, and improvable.