Jika upload putus di 73%, apa state sistem?
Jika object storage sukses tapi DB commit gagal, apa recovery path?
Jika scan result dikirim dua kali, apakah file double-accepted?
Jika delete object gagal setelah metadata DELETION_REQUESTED, siapa retry?
Jika pod mati saat streaming 5GB, apakah heap aman dan file partial dibersihkan?
Jika secret storage expired di tengah upload worker, apa yang terjadi?

upload file

1. Upload session created
2. Payload received in temporary location
3. Payload verified
4. Metadata committed
5. Scan requested
6. Scan decision committed
7. File accepted
8. File deleted/archived

Commit point: metadata status = QUARANTINED
Meaning: payload exists in quarantine storage and passed basic verification.
Recovery: scanner request can be re-enqueued if missing.
User visible: security check in progress.

API streams upload to /tmp/upload-123.part.
Client disconnects.
Partial file remains.

Partial local file must never become accepted payload.

public Path writeToTemporaryFile(InputStream input, Path tempDir, long maxBytes) throws IOException {
    Path tempFile = Files.createTempFile(tempDir, "upload-", ".part");
    long copied = 0;

    try (InputStream in = input;
         OutputStream out = Files.newOutputStream(tempFile,
             StandardOpenOption.WRITE,
             StandardOpenOption.TRUNCATE_EXISTING)) {

        byte[] buffer = new byte[1024 * 1024];
        int read;
        while ((read = in.read(buffer)) != -1) {
            copied += read;
            if (copied > maxBytes) {
                throw new FileTooLargeException(maxBytes);
            }
            out.write(buffer, 0, read);
        }

        out.flush();
        return tempFile;
    } catch (Exception ex) {
        tryDelete(tempFile);
        throw ex;
    }
}

private void tryDelete(Path path) {
    try {
        Files.deleteIfExists(path);
    } catch (IOException cleanupError) {
        // log and let reconciliation cleanup later
    }
}

Incomplete multipart uploads must be aborted or expired.

- store multipart upload ID durably;
- on failure, attempt AbortMultipartUpload;
- run lifecycle rule or cleanup job for abandoned multipart uploads;
- do not create FileArtifact ACCEPTED until multipart complete result is verified.

Object upload succeeds.
DB commit fails.

Payload exists, metadata does not.

Upload object under temporary/incoming key with uploadSessionId.
Only after DB commit and verification can object be promoted.
Incoming prefix has lifecycle cleanup.

public enum RetryDecision {
    RETRY,
    DO_NOT_RETRY,
    REFRESH_SECRET_THEN_RETRY,
    REQUEUE_ASYNC,
    FAIL_FAST
}

public final class FileWorkflowErrorClassifier {
    public RetryDecision classify(Throwable error) {
        if (error instanceof FileTooLargeException) {
            return RetryDecision.DO_NOT_RETRY;
        }
        if (error instanceof InvalidContentTypeException) {
            return RetryDecision.DO_NOT_RETRY;
        }
        if (error instanceof AccessDeniedException) {
            return RetryDecision.FAIL_FAST;
        }
        if (error instanceof TransientStorageException) {
            return RetryDecision.RETRY;
        }
        if (error instanceof SecretExpiredException) {
            return RetryDecision.REFRESH_SECRET_THEN_RETRY;
        }
        if (error instanceof ScannerUnavailableException) {
            return RetryDecision.REQUEUE_ASYNC;
        }
        return RetryDecision.FAIL_FAST;
    }
}

Request thread should do bounded work required to produce a truthful response.
Long, flaky, or external workflows should be represented as lifecycle state and continued asynchronously.

idempotencyKey = random string only

scope = tenantId + actorId + operationType + idempotencyKey

CREATE TABLE idempotency_record (
    scope              VARCHAR(255) NOT NULL,
    idempotency_key    VARCHAR(255) NOT NULL,
    operation_type     VARCHAR(100) NOT NULL,
    request_hash       CHAR(64) NOT NULL,
    response_json      JSONB,
    status             VARCHAR(50) NOT NULL,
    created_at         TIMESTAMP WITH TIME ZONE NOT NULL,
    expires_at         TIMESTAMP WITH TIME ZONE NOT NULL,
    PRIMARY KEY (scope, idempotency_key)
);

public IdempotencyResult check(String scope, String key, String requestHash) {
    IdempotencyRecord existing = repository.find(scope, key);
    if (existing == null) {
        return IdempotencyResult.newOperation();
    }
    if (!existing.requestHash().equals(requestHash)) {
        throw new IdempotencyConflictException(scope, key);
    }
    return IdempotencyResult.replay(existing.response());
}

First call:
- verifies object
- creates/updates metadata
- returns fileId

Second call with same key:
- returns same fileId
- does not create new metadata
- does not enqueue duplicate scan request

public void cleanupObject(String bucket, String key) {
    try {
        objectStorage.deleteIfExists(bucket, key);
        metrics.increment("file_cleanup_object_success_total");
    } catch (TransientStorageException ex) {
        cleanupRepository.enqueue(bucket, key, CleanupReason.TRANSIENT_DELETE_FAILURE);
        metrics.increment("file_cleanup_object_requeued_total");
    }
}

public void expireStaleUploads(Instant now) {
    List<UploadSession> stale = uploadSessionRepository.findExpiredActiveSessions(now);

    for (UploadSession session : stale) {
        try {
            objectStorage.abortOrDeleteTemporaryUpload(session.temporaryStorageKey());
            uploadSessionRepository.markExpired(session.id(), session.version());
            audit.recordUploadExpired(session.id(), session.ownerUserId(), now);
        } catch (ConcurrentModificationException ignored) {
            // another worker handled it
        } catch (Exception ex) {
            metrics.increment("upload_session_expire_failed_total");
            log.warn("Failed to expire uploadSessionId={}", session.id(), ex);
        }
    }
}

If metadata says ACCEPTED and payload missing:
- emit critical alert;
- mark operational incident;
- attempt restore from replica/version/backup;
- do not silently mark deleted;
- do not recreate empty placeholder;
- preserve audit trail.

CREATE TABLE upload_part (
    upload_session_id VARCHAR(64) NOT NULL,
    part_number       INTEGER NOT NULL,
    size_bytes        BIGINT NOT NULL,
    sha256            CHAR(64),
    storage_etag      VARCHAR(255),
    status            VARCHAR(50) NOT NULL,
    uploaded_at       TIMESTAMP WITH TIME ZONE,
    PRIMARY KEY (upload_session_id, part_number)
);

A completed part must have part number, size, storage provider acknowledgement,
and optional checksum/hash depending on provider support.

When downstream storage/scan is unhealthy, stop accepting unlimited uploads.

{
  "error": "java.nio.file.NoSuchFileException: /tmp/upload-123"
}

{
  "errorCode": "UPLOAD_TEMPORARILY_UNAVAILABLE",
  "message": "The file could not be processed right now. Please retry later.",
  "correlationId": "REQ-01JZ..."
}

{
  "fileId": "FILE-01JZ...",
  "status": "SCANNING",
  "downloadAvailable": false
}

scan(file);
repository.markClean(file.id());
storage.promote(file);

SCANNING -> SCANNED_CLEAN -> ACCEPTING -> ACCEPTED

SCANNED_CLEAN -> ACCEPTING -> ACCEPTED

- Local temp file is disposable.
- No correctness-critical state only in local file.
- Temp path must include upload/session ID only if safe and sanitized.
- Cleanup runs on startup and schedule.
- Temp file has max age.
- Temp volume has quota.

@Component
public final class TempDirectoryJanitor implements ApplicationRunner {
    private final Path tempDir;

    @Override
    public void run(ApplicationArguments args) throws IOException {
        Instant cutoff = Instant.now().minus(Duration.ofHours(2));

        try (Stream<Path> paths = Files.list(tempDir)) {
            paths.filter(path -> path.getFileName().toString().endsWith(".part"))
                 .forEach(path -> deleteIfOlderThan(path, cutoff));
        }
    }

    private void deleteIfOlderThan(Path path, Instant cutoff) {
        try {
            FileTime modified = Files.getLastModifiedTime(path);
            if (modified.toInstant().isBefore(cutoff)) {
                Files.deleteIfExists(path);
            }
        } catch (IOException ex) {
            // log, metric, continue
        }
    }
}

- Validate new config candidate.
- If valid, atomically swap effective config.
- If invalid, reject reload and keep previous config.
- Emit metric and alert.

- classify as auth/secret failure;
- refresh credential if supported;
- retry bounded operation;
- if refresh fails, stop worker/readiness degraded;
- do not mark file rejected as if user uploaded invalid file;
- keep lifecycle in retryable operational state.

User file invalid -> domain rejection.
Platform credential invalid -> operational failure.

file_workflow_error_total{stage,error_class,retryable}
file_workflow_retry_total{stage}
file_workflow_compensation_total{type,result}
upload_session_expired_total
upload_temp_cleanup_total
object_orphan_detected_total
object_missing_for_metadata_total
file_reconciliation_run_total
file_reconciliation_failure_total
file_status_age_seconds{status}
file_worker_stuck_total{stage}

upload.receive
upload.stage_temp
upload.object_put
upload.metadata_commit
file.verify
file.scan_request
file.scan_result
file.accept
file.delete

- object missing for ACCEPTED file > 0
- file stuck in ACCEPTING > threshold
- scan backlog age > SLA
- cleanup failures increasing
- storage retry rate high
- secret refresh failure for file workers
- orphan object count spike

1. Is scanner service healthy?
2. Is scan queue backlog growing?
3. Can scanner read quarantine object?
4. Are scanner credentials valid?
5. Are scan result events being delivered?
6. Are DB writes failing?
7. Are specific file types/sizes causing timeouts?

- re-enqueue scan request for affected file IDs;
- increase worker capacity if safe;
- quarantine problematic files;
- pause acceptance if scanner trust boundary is broken;
- communicate user-visible delay.

1. Confirm object key and bucket from metadata.
2. Check object versioning/replica/backup.
3. Check recent lifecycle/delete job activity.
4. Check audit events for deletion or overwrite.
5. Check IAM/storage access denial vs actual absence.
6. Preserve evidence before modifying state.

- restore payload from version/replica/backup;
- block deletion workers if suspected bug;
- mark incident;
- do not silently mark file deleted;
- document audit trail.

1. Are DB commits failing after upload?
2. Did new deployment change object key pattern?
3. Is idempotency broken?
4. Are clients abandoning uploads?
5. Is cleanup job disabled?

- apply lifecycle cleanup for incoming prefix;
- tag uncertain objects;
- delete only objects proven safe;
- fix root cause before mass cleanup.

For all random command sequences:
- DELETED never transitions to ACCEPTED
- ACCEPTED always has checksum and clean scan
- REJECTED is never downloadable
- legal hold file never becomes DELETED

System either preserves invariant or emits actionable alert with recovery path.

- authenticate/authorize
- validate metadata
- create upload session
- stream with max limit
- stage to temp/object storage
- record durable status
- return truthful lifecycle status

- load candidate by status
- acquire via optimistic lock or lease
- perform idempotent external side effect
- verify side effect
- transition state
- write audit/outbox
- retry/compensate on classified failure

- scan for impossible/stale combinations
- avoid destructive action when uncertain
- repair safe cases automatically
- alert critical cases
- record what was repaired