This part does not assume CSG Quote & Order uses Kubernetes CronJob, Quartz,
Spring Scheduler, platform scheduler, DB scheduler, internal workflow engine,
Redis locking, PostgreSQL advisory lock, or a specific batch framework.

Treat all scheduler, locking, reconciliation, and batch execution details as
internal verification items.

expire stale quotes
recalculate pricing snapshots
reconcile order status with downstream system
retry failed external delivery
cleanup temporary files
archive old audit records
publish delayed notifications
repair derived read models
refresh catalog cache
close abandoned checkout/order flows

at a scheduled time
periodically
as a delayed retry
in bulk
as a repair action
as a consistency check
outside a user request

"Just run a script every night."

A job is a distributed state transition mechanism with its own trigger,
lease, work selection, idempotency, progress tracking, observability,
retry, and recovery model.

When should the job start?

What work should be processed?
How much work should be processed?
How is progress tracked?
What happens if the job dies?
What happens if another copy starts?
What is safe to retry?

CronJob schedule
  -> Job object
    -> Pod
      -> container process
        -> Java main/job runner
          -> business job logic

apiVersion: batch/v1
kind: CronJob
metadata:
  name: quote-expiration-reconciliation
spec:
  schedule: "*/15 * * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 1
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: job
              image: example-registry/quote-service:1.2.3
              args:
                - "run-job"
                - "quote-expiration-reconciliation"
              envFrom:
                - configMapRef:
                    name: quote-service-config
                - secretRef:
                    name: quote-service-secret

cluster failover can produce edge cases
manual run can overlap
another environment can target same DB
job can partially commit then die
operator can retry a failed Job

uses heavy CPU/DB resources
runs long
has a different scaling model
needs different permissions
must be triggered manually by operators
has high blast radius

execution is guarded by a robust lease
work is lightweight
failure is observable
all replicas coordinate correctly
shutdown is graceful

select quotes where expires_at < now
for each quote:
  set status = EXPIRED
  publish QuoteExpired event

DB update succeeds
process dies before event publish
rerun publishes duplicate or misses event depending implementation

select eligible quotes where status = ACTIVE and expires_at < now
for each quote:
  transition ACTIVE -> EXPIRED conditionally
  insert outbox event with unique business key
  commit transaction
publisher later emits event from outbox

public final class QuoteExpirationJob {
    public JobResult run(JobContext context) {
        JobLease lease = leaseService.acquire("quote-expiration", Duration.ofMinutes(10));
        if (!lease.acquired()) {
            return JobResult.skipped("lease-not-acquired");
        }

        try {
            int processed = 0;

            while (processed < context.maxItems()) {
                List<QuoteId> batch = quoteRepository.findExpirableQuotes(context.batchSize());
                if (batch.isEmpty()) break;

                for (QuoteId quoteId : batch) {
                    quoteService.expireIfStillActive(quoteId, context.clock());
                    processed++;
                }
            }

            return JobResult.completed(processed);
        } finally {
            lease.release();
        }
    }
}

SELECT * FROM quote WHERE status = 'ACTIVE' AND expires_at < now();

SELECT id
FROM quote
WHERE status = 'ACTIVE'
  AND expires_at < now()
ORDER BY expires_at, id
LIMIT :batch_size;

SELECT id
FROM pending_job_item
WHERE status = 'READY'
  AND available_at <= now()
ORDER BY available_at, id
LIMIT :batch_size
FOR UPDATE SKIP LOCKED;

Is ordering still important?
Can work be processed out of order?
What happens if a transaction holds locks too long?
How is a stuck item recovered?

job_name
owner_id
lease_token
acquired_at
expires_at
heartbeat_at
fencing_token if external side effects require it

CREATE TABLE job_lease (
    job_name        text PRIMARY KEY,
    owner_id        text NOT NULL,
    lease_token     text NOT NULL,
    fencing_token   bigint NOT NULL,
    acquired_at     timestamptz NOT NULL,
    heartbeat_at    timestamptz NOT NULL,
    expires_at      timestamptz NOT NULL
);

UPDATE job_lease
SET owner_id = :owner_id,
    lease_token = :lease_token,
    fencing_token = fencing_token + 1,
    acquired_at = now(),
    heartbeat_at = now(),
    expires_at = now() + interval '10 minutes'
WHERE job_name = :job_name
  AND expires_at < now();

status per item
checkpoint cursor
watermark timestamp
run record
outbox/inbox dedupe record
business-state conditional update

CREATE TABLE job_run (
    id              uuid PRIMARY KEY,
    job_name        text NOT NULL,
    trigger_type    text NOT NULL,
    started_at      timestamptz NOT NULL,
    finished_at     timestamptz,
    status          text NOT NULL,
    owner_id        text NOT NULL,
    processed_count bigint NOT NULL DEFAULT 0,
    failed_count    bigint NOT NULL DEFAULT 0,
    last_error_code text,
    last_error_text text
);

attempt count
next_attempt_at
last_error_code
last_error_message
retry classification
dead-letter/final failure state
manual repair option

CREATE TABLE delivery_attempt_item (
    id              uuid PRIMARY KEY,
    business_key    text NOT NULL,
    status          text NOT NULL,
    attempt_count   int NOT NULL DEFAULT 0,
    next_attempt_at timestamptz NOT NULL,
    last_error_code text,
    last_error_at   timestamptz,
    created_at      timestamptz NOT NULL,
    updated_at      timestamptz NOT NULL,
    UNIQUE (business_key)
);

what data is eligible
what retention rule applies
whether data has legal/compliance value
whether data can be restored
how deletion is audited
how dry-run works

DELETE FROM temp_file WHERE created_at < now() - interval '1 day';

SELECT id
FROM temp_file
WHERE created_at < now() - interval '7 days'
  AND status IN ('COMPLETED', 'FAILED_TERMINAL')
  AND legal_hold = false
ORDER BY created_at, id
LIMIT :batch_size;

mode=dry-run
  -> select eligible records
  -> log and metric expected deletion count
  -> do not mutate

mode=execute
  -> mutate in small chunks
  -> record job_run
  -> emit audit summary

API database and Kafka events
source table and read model
quote/order state and workflow engine
internal DB and external downstream system
object storage and metadata table
cache and source of truth
billing state and order state

source of truth
comparison rule
repair action

Can it be run multiple times?
Can it be paused?
Can it resume from checkpoint?
Can it be throttled?
Can it run while old and new application versions coexist?
Can it be rolled forward if rollback is impossible?
Does it emit progress metrics?
Does it have a dry-run estimate?

select next N records by stable cursor
process transactionally
record checkpoint
sleep/throttle if needed
repeat

SELECT id
FROM quote
WHERE id > :last_seen_id
ORDER BY id
LIMIT :batch_size;

job_started_total
job_completed_total
job_failed_total
job_skipped_total
job_duration_seconds
job_items_processed_total
job_items_failed_total
job_lag_seconds
job_last_success_timestamp
job_lock_acquire_failed_total
job_retry_scheduled_total
job_deadletter_total

job_name
status
error_class

tenant_id with high cardinality
quote_id
order_id
customer_id
raw error message

job_run_id
job_name
trigger_type
owner_id
batch_number
processed_count
failed_count
correlation_id if manually triggered

Did the job run?
Did it finish?
How long did it take?
How many items did it process?
Is backlog growing?
Are failures concentrated by type?
When was the last successful run?

job name
mode: dry-run or execute
scope: tenant/environment/range
reason
requester identity
approval if high risk
correlation ID
max items / throttle

java -jar quote-service-jobs.jar \
  run quote-expiration-reconciliation \
  --mode=dry-run \
  --tenant=tenant-a \
  --max-items=1000 \
  --reason="verify post-migration quote state"

job_run_id=...
requester=...
mode=dry-run
scope=tenant-a
eligible_count=842
mutated_count=0

POST /internal/jobs/{jobName}/runs
GET  /internal/jobs/{jobName}/runs/{runId}
POST /internal/jobs/{jobName}/runs/{runId}/cancel
GET  /internal/jobs/{jobName}/status

POST /expire-all-quotes

validate request
authorize operator/system identity
create job_run record
enqueue or signal execution
return 202 Accepted with run ID
job executes out-of-band
client polls or receives event

HTTP/1.1 202 Accepted
Location: /internal/jobs/quote-expiration/runs/5ad0...
Content-Type: application/json

{
  "jobRunId": "5ad0...",
  "status": "ACCEPTED",
  "mode": "DRY_RUN"
}

[ ] What scheduler is used: Kubernetes CronJob, Quartz, platform scheduler, workflow engine, manual job runner, or in-process scheduler?
[ ] Are jobs deployed as separate workloads or inside the API service?
[ ] Is there a standard job_run table or operational tracking model?
[ ] Is there a platform standard for job locking/lease?
[ ] Are PostgreSQL advisory locks, Redis locks, or DB row leases used?
[ ] Is there a standard retry/DLQ model for job items?
[ ] Are job metrics standardized?
[ ] Are job alerts configured for stale success, failure rate, backlog, and duration?
[ ] Is manual job triggering allowed? Through what tool/API?
[ ] Are manual job runs audited?
[ ] Are jobs tenant-aware?
[ ] Are cleanup/archive jobs reviewed for retention/compliance?
[ ] Are reconciliation jobs allowed to auto-repair or only report?
[ ] Are backfill jobs reviewed as part of release plan?
[ ] Are job pods given separate CPU/memory limits from online API services?
[ ] Are job secrets/permissions more limited than API service permissions?
[ ] Are failed jobs recoverable without direct DB surgery?

[ ] What is the trigger?
[ ] What is the unit of work?
[ ] Is processing idempotent?
[ ] Can the job safely run twice?
[ ] What prevents overlap?
[ ] What happens if the process dies mid-batch?
[ ] Is progress persisted?
[ ] Is work selected in bounded chunks?
[ ] Is there a dry-run mode for dangerous jobs?
[ ] Is the job tenant-aware?
[ ] Are transactions small enough?
[ ] Are external calls retried with budget/backoff?
[ ] Are poison items isolated?
[ ] Is there a terminal failure state?
[ ] Are metrics and logs sufficient?
[ ] Is there an alert if the job stops running?
[ ] Is manual operation documented?
[ ] Does the release plan account for this job?

ownership
contract
idempotency
observability
security
tenant isolation
resource limits
recovery model
change management

Does the cron expression work?

Can this job run twice, fail halfway, recover safely, prove what it did,
and avoid damaging customer/business state?