This part does not assume CSG uses AWS, Azure, S3, Blob Storage, Parameter Store, AppConfig, Secrets Manager, Azure App Configuration, or Key Vault.
Treat every cloud service and SDK integration as an internal verification item.

Java code
-> SDK client
-> credential provider
-> request signing / token acquisition
-> DNS
-> network path
-> TLS
-> cloud endpoint / private endpoint
-> IAM/RBAC policy
-> service quota
-> retry/throttle behavior
-> audit log

Where do credentials come from?
What identity does the pod/process use?
Which region/account/subscription/tenant is called?
Is the endpoint public or private?
What is the timeout/retry policy?
What happens during throttling?
Can secret rotation happen without redeploy?
How do we avoid logging secrets?
How do we test locally without production credentials?

A cloud SDK client must be configured like any other outbound dependency:
timeout, retry, identity, endpoint, region, metrics, tracing, error mapping, and ownership.

@GET
@Path("/documents/{id}")
public Response download(@PathParam("id") String id) {
    S3Client s3 = S3Client.builder().build(); // bad: per request client creation
    // ...
}

repeated client initialization
uncontrolled connection pools
credential provider churn
harder shutdown
harder metrics/tracing
harder test injection

public final class ObjectStorageClientProvider {
    private final ObjectStorageClient client;

    public ObjectStorageClientProvider(ObjectStorageConfig config, CredentialsProvider credentials) {
        this.client = ObjectStorageClient.create(config, credentials);
    }

    public ObjectStorageClient client() {
        return client;
    }
}

Cloud SDK clients are infrastructure resources.
Construct once per config scope, inject, reuse, close on shutdown if required.

region
credentials provider
service client
HTTP client implementation
retry policy
timeout policy
endpoint override if needed
request/response transformer for streaming

Prefer temporary workload identity over static access keys.

credential/token provider
resource endpoint
client builder
service-specific client
retry/timeout policy
Azure tenant/subscription/resource group context
RBAC role assignment
private endpoint/DNS if configured

Prefer managed identity / workload identity over connection strings and static secrets.

What credential provider is active in local development?
What provider is active in Kubernetes?
What provider is active in CI?
What provider is active in on-prem deployment?
Could the SDK silently pick the wrong credentials?
Could local credentials accidentally access production?

Authentication success is not authorization correctness.
The identity may be valid but still wrong.

bucket/container
-> object/blob key
-> metadata
-> content stream
-> versioning/lifecycle policy maybe
-> access policy/IAM/RBAC/SAS/presigned URL maybe

large quote/order attachments
export files
import files
generated documents
audit artifacts
integration payload archive
reconciliation reports

Who owns object key naming?
Is object immutable after write?
Is versioning enabled?
What metadata is required?
What retention policy applies?
Is object encrypted?
How are large downloads streamed?
How are partial failures reconciled?

tenant/{tenantId}/quote/{quoteId}/attachment/{attachmentId}/{filename}

tenant-hash/{tenantBucket}/object/{objectId}
metadata table maps objectId -> business context

Object storage operations need reconciliation because object storage and database update are not one atomic transaction.

authorization before access
streaming response
range request if supported/required
content-disposition
content-type
checksum/eTag
cache control
rate limiting
large file timeout
client disconnect

@GET
@Path("/documents/{objectId}/content")
public Response download(@PathParam("objectId") String objectId) {
    // authorize business access first
    // resolve metadata from DB
    // open object stream from storage
    // return StreamingOutput / response body stream
    // close stream correctly
    return Response.ok(streamingOutput)
        .type(metadata.contentType())
        .header("Content-Disposition", metadata.contentDisposition())
        .build();
}

read full object into byte[] for large files
log object content
trust object key from user input
skip tenant authorization because object id is opaque

AWS Systems Manager Parameter Store
AWS AppConfig
Azure App Configuration
internal config service
Kubernetes ConfigMap / external secret operator

unclear precedence
runtime drift across pods
partial rollout inconsistency
invalid config loaded dynamically
secret value treated as normal config
missing audit trail

Dynamic config must be validated as strictly as startup config.

application default
-> environment profile config
-> Kubernetes ConfigMap
-> cloud config store
-> tenant-specific config
-> emergency override / kill switch

If the same key exists in three places, which wins?
Is the winning source visible in diagnostics?
Can we detect drift between pods?
Can runtime reload happen safely?
Can config be rolled back?

/config/diagnostics
- key name
- effective source
- sanitized value or hash
- last loaded time
- version/etag

passwords
API keys
private keys
client secrets
certificates
signing material
credential references

never log secret values
never put secret in exception messages
never expose secret through config diagnostics
prefer references over raw values in normal config
rotate secrets with overlap window
monitor secret access failures

create new secret version
-> allow both old and new where possible
-> deploy/reload consumers
-> verify new credential is used
-> revoke old credential
-> monitor failures

Can this secret be rotated without emergency code change?

network timeout
TLS/DNS failure
credential/token failure
IAM/RBAC deny
service throttling
quota limit
regional outage
request too large
object not found
conditional write conflict

max total time spent retrying must fit caller timeout and business SLA

HTTP request timeout: 5s
SDK retry policy worst-case: 30s

SDK timeout + retry budget < resource method timeout < gateway timeout

only write if object does not exist
only update if ETag matches
only use expected version
only promote config version after validation

service A reads config version 10
service B updates to version 11
service A writes based on stale version 10
without condition, B's update can be overwritten

If the resource is mutable and shared, require optimistic concurrency or explicit ownership.

pod subnet
network policy
service account / identity
DNS resolver
private endpoint DNS zone
VPC/VNet route table
security group / NSG
proxy if any
TLS trust store
cloud service endpoint

For cloud SDK issues, always separate auth failure, authorization failure, DNS failure, network failure, and service error.

dev/test/stage/prod
multiple AWS accounts
multiple Azure subscriptions
multiple regions
multiple tenants
hybrid on-prem/cloud

explicit region config
explicit account/subscription expectation
startup validation of cloud identity
resource name includes environment only if naming policy allows
no production credentials in local dev
separate IAM/RBAC per environment

expectedEnvironment=prod
expectedCloudAccount=123456789012
actualCallerIdentity=123456789012
expectedRegion=ap-southeast-1
configuredBucket=csg-prod-quote-documents

local emulator where practical
mock object storage adapter
dev cloud account with restricted resources
fake secret provider
Docker Compose with MinIO/Azurite if aligned internally
recorded test fixtures

IAM/RBAC
audit logging
private endpoints
large object performance
service quotas
conditional request semantics
retry/throttling behavior

Local convenience must not train engineers to bypass production identity and security boundaries.

object upload succeeds
object upload fails after DB metadata created
DB update fails after object upload
secret unavailable during startup
secret rotates while service is running
credential provider cannot resolve identity
wrong region/account/subscription configured
cloud service returns 429/503

service name
operation
region
account/subscription indicator if allowed
endpoint type: public/private/internal
status/error code
latency histogram
retry count
throttle count
request size / response size bucket
credential provider type, sanitized

object key
customer id
full tenant id if high cardinality
secret name if sensitive
raw endpoint if it contains env/customer data

{
  "event": "cloud_object_upload_failed",
  "provider": "aws",
  "service": "s3",
  "operation": "putObject",
  "objectId": "obj_123",
  "businessType": "quote_attachment",
  "errorCategory": "throttled",
  "retryable": true,
  "correlationId": "..."
}

secret value
access key
session token
SAS token
presigned URL with signature
full object content
raw authorization header

Can this service read only its required bucket/container/path?
Can it write but not delete?
Can it read only specific secret names?
Can it access only required config labels/environments?
Can local/dev identity access prod?
Are audit logs enabled for sensitive operations?

Application runtime identity should rarely have permission to change its own production security baseline.

[ ] Is AWS used, Azure used, both, or neither?
[ ] Which SDK versions are used?
[ ] Are SDK clients wrapped behind internal abstractions?
[ ] Are clients sync or async?
[ ] Which HTTP client implementation is configured?
[ ] Where are timeout/retry policies configured?

[ ] What identity does the service use in Kubernetes/cloud/on-prem?
[ ] Does it use static secrets, workload identity, managed identity, IRSA, or another platform mechanism?
[ ] Is credential resolution explicit enough to avoid wrong identity?
[ ] Are local dev credentials isolated from production?
[ ] Is startup identity validation implemented?

[ ] Is S3, Azure Blob, both, or another object store used?
[ ] What buckets/containers exist per environment/tenant?
[ ] What object key naming policy exists?
[ ] Is versioning enabled?
[ ] Are checksums/eTags used?
[ ] Is there orphan object reconciliation?
[ ] Are large files streamed?

[ ] Is Parameter Store/AppConfig/Azure App Configuration/internal config service used?
[ ] What is config precedence?
[ ] Does runtime reload exist?
[ ] How is invalid config rejected?
[ ] Is config drift detectable per pod?
[ ] Are config changes audited?

[ ] Is Secrets Manager/Key Vault/internal vault used?
[ ] Are secret values cached? With what TTL?
[ ] How does rotation happen?
[ ] Is there overlap between old and new secret versions?
[ ] Are secrets redacted from logs/errors/diagnostics?
[ ] Are access failures alerted?

[ ] Are cloud services reached through public endpoints or private endpoints?
[ ] Are DNS private zones configured?
[ ] Are network policies/security groups/NSGs documented?
[ ] Is TLS inspection/proxy involved?
[ ] Is there a runbook for DNS/TLS/403/timeout failure separation?

[ ] Is the SDK client lifecycle correct and reusable?
[ ] Are timeout and retry budgets explicit?
[ ] Is the operation idempotent or protected by condition/idempotency key?
[ ] Is object storage coordinated with DB using reconciliation?
[ ] Are large objects streamed instead of buffered?
[ ] Are credentials sourced from approved provider?
[ ] Is least privilege preserved?
[ ] Are secrets never logged or exposed through diagnostics?
[ ] Is region/account/subscription explicit and validated?
[ ] Is config precedence documented?
[ ] Does secret/config rotation work without unsafe redeploy?
[ ] Are metrics/logs/traces sufficient for cloud-call debugging?
[ ] Are local tests not dependent on production credentials?
[ ] Is private endpoint/DNS behavior considered?

identity is part of the contract
region/account/subscription is part of the contract
retry/timeout is part of the contract
object key naming is part of the contract
secret rotation is part of the contract
network path is part of the contract
observability is part of the contract

If credentials rotate tonight, will the service survive?
If object upload succeeds but DB update fails, can we reconcile?
If DNS/private endpoint breaks, can we diagnose without guessing?
If the SDK retries for 30 seconds, does it violate the caller's SLA?
If a developer runs locally, can they avoid touching production resources?