billing-worker calls invoice-api
order-service calls pricing-service
nightly-job calls report-api
integration-adapter calls partner-api

Client authenticates as itself.
Authorization server issues access token for that client.
Resource server authorizes based on client identity, scope, audience, and policy.

service credential -> token endpoint -> short-lived access token -> protected API

User clicked approve order.
User created case.
User requested personal data export.

Client credentials answers: which service is calling?
It does not answer: which human initiated this action?

POST /oauth2/token HTTP/1.1
Host: idp.example.com
Content-Type: application/x-www-form-urlencoded
Authorization: Basic base64(client_id:client_secret)

grant_type=client_credentials&scope=pricing:read

{
  "access_token": "eyJ...",
  "token_type": "Bearer",
  "expires_in": 300,
  "scope": "pricing:read"
}

ServiceClient
  id
  client_id
  display_name
  owner_team
  environment
  tenant_id? / organization_id?
  allowed_grants
  allowed_scopes
  allowed_audiences
  auth_method
  status
  created_at
  rotated_at

ClientCredential
  id
  client_id
  credential_type
  secret_hash / jwk_thumbprint / cert_thumbprint
  active_from
  expires_at
  status

ServiceAccountPolicy
  client_id
  resource
  action/scope
  conditions

small/internal: client_secret_basic may be acceptable
large/high-risk: private_key_jwt or mTLS
cloud-native: workload identity federation where available

Store in secret manager.
Inject at runtime.
Do not bake into image.
Do not log.
Do not commit.
Rotate periodically.
Support overlapping old/new secret.
Limit blast radius per client/environment.

client-secret: orders-secret-123

client-secret: ${ORDERS_CLIENT_SECRET}

runtime secret reference from Vault / AWS Secrets Manager / Azure Key Vault / Kubernetes Secret sealed/encrypted pipeline

{
  "iss": "orders-worker",
  "sub": "orders-worker",
  "aud": "https://idp.example.com/oauth2/token",
  "jti": "random-one-time-id",
  "exp": 1783078800,
  "iat": 1783078500
}

public String clientAssertion(PrivateKey key, String keyId, String clientId, URI tokenEndpoint) {
    Instant now = Instant.now();

    return Jwts.builder()
        .issuer(clientId)
        .subject(clientId)
        .audience().add(tokenEndpoint.toString()).and()
        .id(UUID.randomUUID().toString())
        .issuedAt(Date.from(now))
        .expiration(Date.from(now.plusSeconds(300)))
        .signWith(key, Jwts.SIG.RS256)
        .header().keyId(keyId).and()
        .compact();
}

Client proves possession of private key during TLS handshake.
Authorization server maps certificate to OAuth client.
Issued token can be bound to certificate.
Resource server rejects token unless caller presents same certificate.

scope=api
scope=read
scope=admin

pricing:read
pricing:quote
orders:create
orders:status:read
case:assignment:write

Who is this token intended for?

What operation category is allowed?

Who is the caller?

issuer == trusted_idp
audience contains pricing-api
client_id == orders-service
scope contains pricing:quote
client status == active

{
  "iss": "https://idp.example.com/realms/acme",
  "sub": "client:orders-service",
  "aud": "pricing-api",
  "azp": "orders-service",
  "client_id": "orders-service",
  "scope": "pricing:read pricing:quote",
  "exp": 1783078800,
  "iat": 1783078500,
  "jti": "..."
}

Resource server must extract service identity from a provider-specific, validated, documented claim mapping.

5 to 15 minutes for normal service calls
1 to 5 minutes for high-risk APIs
longer only with explicit risk acceptance

Every outbound request calls /token first.

Cache token by client registration + audience + scope set.
Refresh before expiry with jitter.
Use single-flight refresh to avoid stampede.

record TokenCacheKey(String registrationId, String audience, Set<String> scopes) {}

record CachedToken(String value, Instant expiresAt) {
    boolean isUsable(Clock clock) {
        return Instant.now(clock).isBefore(expiresAt.minusSeconds(30));
    }
}

public CompletionStage<CachedToken> token(TokenCacheKey key) {
    CachedToken cached = cache.get(key);
    if (cached != null && cached.isUsable(clock)) {
        return CompletableFuture.completedFuture(cached);
    }
    return inFlight.computeIfAbsent(key, this::fetchAndCache)
        .whenComplete((t, e) -> inFlight.remove(key));
}

spring:
  security:
    oauth2:
      client:
        registration:
          pricing:
            provider: keycloak
            client-id: orders-service
            client-secret: ${ORDERS_SERVICE_CLIENT_SECRET}
            authorization-grant-type: client_credentials
            scope:
              - pricing:read
              - pricing:quote
        provider:
          keycloak:
            issuer-uri: https://idp.example.com/realms/acme

@Configuration
class OAuthClientCredentialsConfig {

    @Bean
    OAuth2AuthorizedClientManager authorizedClientManager(
        ClientRegistrationRepository clientRegistrationRepository,
        OAuth2AuthorizedClientService authorizedClientService
    ) {
        OAuth2AuthorizedClientProvider provider = OAuth2AuthorizedClientProviderBuilder.builder()
            .clientCredentials()
            .build();

        AuthorizedClientServiceOAuth2AuthorizedClientManager manager =
            new AuthorizedClientServiceOAuth2AuthorizedClientManager(
                clientRegistrationRepository,
                authorizedClientService
            );

        manager.setAuthorizedClientProvider(provider);
        return manager;
    }
}

@Service
final class PricingTokenProvider {

    private final OAuth2AuthorizedClientManager manager;

    PricingTokenProvider(OAuth2AuthorizedClientManager manager) {
        this.manager = manager;
    }

    String accessToken() {
        OAuth2AuthorizeRequest request = OAuth2AuthorizeRequest
            .withClientRegistrationId("pricing")
            .principal("orders-service")
            .build();

        OAuth2AuthorizedClient client = manager.authorize(request);
        if (client == null || client.getAccessToken() == null) {
            throw new IllegalStateException("Unable to authorize pricing client");
        }
        return client.getAccessToken().getTokenValue();
    }
}

@Configuration
class PricingWebClientConfig {

    @Bean
    WebClient pricingWebClient(
        OAuth2AuthorizedClientManager authorizedClientManager
    ) {
        ServletOAuth2AuthorizedClientExchangeFilterFunction oauth =
            new ServletOAuth2AuthorizedClientExchangeFilterFunction(authorizedClientManager);

        oauth.setDefaultClientRegistrationId("pricing");

        return WebClient.builder()
            .baseUrl("https://pricing-api.internal")
            .apply(oauth.oauth2Configuration())
            .build();
    }
}

@Service
class PricingClient {

    private final WebClient pricingWebClient;

    PricingClient(WebClient pricingWebClient) {
        this.pricingWebClient = pricingWebClient;
    }

    Mono<PriceQuote> quote(QuoteRequest request) {
        return pricingWebClient.post()
            .uri("/quotes")
            .bodyValue(request)
            .retrieve()
            .bodyToMono(PriceQuote.class);
    }
}

Default client registration is convenient.
It can be dangerous if the same WebClient is reused for multiple audiences.

@Service
class PricingRestClient {

    private final RestClient restClient;
    private final PricingTokenProvider tokens;

    PricingRestClient(RestClient.Builder builder, PricingTokenProvider tokens) {
        this.restClient = builder
            .baseUrl("https://pricing-api.internal")
            .build();
        this.tokens = tokens;
    }

    PriceQuote quote(QuoteRequest request) {
        return restClient.post()
            .uri("/quotes")
            .header(HttpHeaders.AUTHORIZATION, "Bearer " + tokens.accessToken())
            .body(request)
            .retrieve()
            .body(PriceQuote.class);
    }
}

@Provider
public class OAuthClientCredentialsFilter implements ClientRequestFilter {

    private final MachineTokenProvider tokenProvider;

    public OAuthClientCredentialsFilter(MachineTokenProvider tokenProvider) {
        this.tokenProvider = tokenProvider;
    }

    @Override
    public void filter(ClientRequestContext requestContext) {
        MachineToken token = tokenProvider.tokenFor("pricing-api", Set.of("pricing:quote"));
        requestContext.getHeaders().putSingle(
            HttpHeaders.AUTHORIZATION,
            "Bearer " + token.value()
        );
    }
}

@Configuration
class PricingApiSecurityConfig {

    @Bean
    SecurityFilterChain api(HttpSecurity http) throws Exception {
        return http
            .authorizeHttpRequests(auth -> auth
                .requestMatchers(HttpMethod.POST, "/quotes")
                    .access(hasClientScope("orders-service", "pricing:quote"))
                .anyRequest().authenticated()
            )
            .oauth2ResourceServer(oauth -> oauth.jwt(jwt -> jwt
                .jwtAuthenticationConverter(machineJwtConverter())
            ))
            .build();
    }

    private AuthorizationManager<RequestAuthorizationContext> hasClientScope(
        String clientId,
        String scope
    ) {
        return (authentication, context) -> {
            Authentication auth = authentication.get();
            boolean allowed = auth.getName().equals(clientId)
                && auth.getAuthorities().stream()
                    .anyMatch(a -> a.getAuthority().equals("SCOPE_" + scope));
            return new AuthorizationDecision(allowed);
        };
    }

    private Converter<Jwt, ? extends AbstractAuthenticationToken> machineJwtConverter() {
        return jwt -> {
            String clientId = Optional.ofNullable(jwt.getClaimAsString("client_id"))
                .orElse(jwt.getClaimAsString("azp"));

            Collection<GrantedAuthority> authorities = extractScopes(jwt);
            return new JwtAuthenticationToken(jwt, authorities, clientId);
        };
    }

    private Collection<GrantedAuthority> extractScopes(Jwt jwt) {
        String scope = jwt.getClaimAsString("scope");
        if (scope == null || scope.isBlank()) return List.of();
        return Arrays.stream(scope.split(" "))
            .map(s -> new SimpleGrantedAuthority("SCOPE_" + s))
            .toList();
    }
}

@Bean
JwtDecoder jwtDecoder(OAuth2ResourceServerProperties properties) {
    NimbusJwtDecoder decoder = JwtDecoders.fromIssuerLocation(properties.getJwt().getIssuerUri());

    OAuth2TokenValidator<Jwt> audienceValidator = jwt -> {
        return jwt.getAudience().contains("pricing-api")
            ? OAuth2TokenValidatorResult.success()
            : OAuth2TokenValidatorResult.failure(new OAuth2Error("invalid_token", "Missing audience", null));
    };

    decoder.setJwtValidator(new DelegatingOAuth2TokenValidator<>(
        JwtValidators.createDefaultWithIssuer(properties.getJwt().getIssuerUri()),
        audienceValidator
    ));

    return decoder;
}

issuer
client_id / azp / sub
scope
resource/audience
environment
tenant
route/method
business context

create table service_access_policy (
    id uuid primary key,
    resource_service text not null,
    caller_client_id text not null,
    required_scope text not null,
    http_method text not null,
    path_pattern text not null,
    tenant_id uuid,
    enabled boolean not null default true,
    created_at timestamptz not null,
    unique (resource_service, caller_client_id, required_scope, http_method, path_pattern, tenant_id)
);

Is the client global, tenant-scoped, or environment-scoped?

{
  "client_id": "orders-service-prod",
  "aud": "pricing-api-prod",
  "tenant": "acme",
  "scope": "pricing:quote"
}

dev
staging
production

Separate issuer or realm per environment.
Separate client registration per environment.
Separate audience per environment.
Separate secrets/keys per environment.
Reject tokens from non-prod issuer in prod.

1. Create new credential while old remains valid.
2. Deploy new secret to service.
3. Verify token acquisition with new secret.
4. Monitor old secret usage.
5. Disable old secret after rollout window.
6. Alert if old secret is used after retirement.

publish new public key
start signing with new key id
keep old public key until old assertions expire
remove old key

Cache tokens until near expiry.
Refresh early with jitter.
Use circuit breaker for token endpoint.
Avoid synchronized refresh across pods.
Expose token acquisition metrics.
Fail closed for protected calls.

oauth_token_request_count
oauth_token_request_latency
oauth_token_request_failure_count
oauth_token_cache_hit_ratio
oauth_token_refresh_in_flight
oauth_token_seconds_until_expiry
oauth_token_invalid_client_count

100 pods start at same time.
All obtain token expiring at same time.
All refresh at same time.
Authorization server rate-limits them.
Downstream calls fail.

jitter refresh threshold
single-flight per pod
distributed lock only if necessary
stagger deployments
cache per scope/audience

boolean shouldRefresh(CachedToken token) {
    long jitter = ThreadLocalRandom.current().nextLong(15, 60);
    return Instant.now(clock).isAfter(token.expiresAt().minusSeconds(60 + jitter));
}

AWS IAM role -> token exchange
Azure Managed Identity -> token acquisition
GCP Workload Identity -> federated token
Kubernetes service account -> projected token -> federation
SPIFFE ID -> mTLS identity -> OAuth token

runtime identity proves workload
authorization server exchanges/asserts identity
resource server receives OAuth access token

Nightly job uses admin user's refresh token.

nightly-report-worker
scope=reports:generate
resource=report-api
policy=only scheduled report operations

{
  "service_client": "report-worker",
  "business_actor": "system:scheduler",
  "job_id": "...",
  "reason": "monthly-statement-generation"
}

orders-service calls case-api with client token and header X-User-Id: alice

Token exchange / on-behalf-of token.
Signed actor context.
Explicit service action model.
Resource server validates both service and actor.

A resource server must know whether a request is service-owned, user-owned, or service-on-behalf-of-user.

frontend user token -> orders-service
orders-service needs pricing-api token with narrower audience
orders-service exchanges incoming token for downstream token

operation is owned by service itself.
no user-specific authorization is needed.
service policy is sufficient.

operation is initiated by user.
downstream must enforce user authorization.
audit must preserve user subject.

{
  "event": "api_access_granted",
  "resource_service": "pricing-api",
  "caller_client_id": "orders-service",
  "issuer": "https://idp.example.com/realms/prod",
  "audience": "pricing-api",
  "scope": ["pricing:quote"],
  "method": "POST",
  "path": "/quotes",
  "request_id": "...",
  "decision": "allow"
}

{
  "event": "api_access_denied",
  "reason": "missing_scope",
  "caller_client_id": "orders-service",
  "required_scope": "pricing:quote"
}

- token cache hit ratio
- token acquisition latency
- token acquisition error by error code
- calls by downstream audience
- 401/403 rate from downstream
- secret rotation success/failure

- token requests by client_id
- invalid_client count
- invalid_scope count
- rate limited clients
- credential usage by key id / secret version

- accepted/rejected tokens by issuer/audience/client
- missing scope decisions
- expired token decisions
- unknown kid/JWKS failures

Token endpoint over HTTPS.
Resource API over HTTPS/mTLS where appropriate.
No token over plaintext internal network.
No token in query parameter.
No token in logs.
No token forwarded to unrelated service.

Ensure API is not reachable except through gateway.
Strip inbound identity headers at gateway.
Sign forwarded identity or use mTLS.
Audit gateway decision.

POST /oauth2/introspect HTTP/1.1
Authorization: Basic base64(resource_server:secret)
Content-Type: application/x-www-form-urlencoded

token=2YotnFZFEjr1zCsicMWpAA

{
  "active": true,
  "client_id": "orders-service",
  "scope": "pricing:quote",
  "aud": "pricing-api",
  "exp": 1783078800
}

Cache positive introspection only briefly.
Do not cache negative results too long.
Protect introspection client credential.
Fail closed when introspection is unavailable unless policy says otherwise.

1. What service identity is represented by this client?
2. Who owns the client registration?
3. Which environment is it valid in?
4. Which audiences can receive tokens for it?
5. Which scopes are allowed?
6. How is the credential stored?
7. How is it rotated?
8. How long do tokens live?
9. Is token caching implemented safely?
10. Does downstream API validate issuer, audience, expiry, and scope?
11. Does this operation need user accountability?
12. How is abuse detected?
13. What happens if token endpoint is down?
14. What happens if the secret leaks?
15. How do we revoke this service immediately?

- one client per service/environment
- secret/private key from secret manager
- short-lived access token
- token cache with jitter
- explicit audience
- granular scopes
- resource server validates issuer/audience/scope/client
- no raw token logs
- credential rotation runbook
- token endpoint SLO and alerts

[ ] Client registration is service/environment-specific.
[ ] Credential is not in source code or container image.
[ ] Token is cached by audience/scope.
[ ] Refresh uses jitter.
[ ] Token endpoint failures are observable.
[ ] Access token is never logged.
[ ] Separate clients are used for separate audiences where appropriate.

[ ] Client credentials grant enabled only for machine clients.
[ ] Allowed scopes are minimal.
[ ] Token lifetime is short.
[ ] Credential rotation is supported.
[ ] invalid_client events are monitored.
[ ] Token audience is explicit.

[ ] Validates issuer.
[ ] Validates expiry.
[ ] Validates audience.
[ ] Validates scope.
[ ] Extracts client identity correctly.
[ ] Distinguishes service calls from user calls.
[ ] Denies unknown clients.
[ ] Logs allow/deny decisions without token material.

@Test
void rejectsWrongAudience() {
    Jwt jwt = jwtBuilder()
        .issuer("https://idp.example.com/realms/prod")
        .audience(List.of("orders-api"))
        .claim("client_id", "orders-service")
        .claim("scope", "pricing:quote")
        .build();

    OAuth2TokenValidatorResult result = pricingAudienceValidator.validate(jwt);

    assertThat(result.hasErrors()).isTrue();
}

1. Identify affected client_id, environment, scopes, audiences.
2. Check token issuance logs for unusual activity.
3. Create replacement credential.
4. Deploy replacement credential.
5. Disable leaked credential.
6. Revoke active tokens if possible.
7. Increase monitoring for affected client.
8. Review access logs for resource servers.
9. Document blast radius and remediation.
10. Fix root cause: repo leak, log leak, CI leak, secret access policy.

Short lifetime limits blast radius.
Emergency key/issuer/client disable may be required.
Resource servers can block affected client_id temporarily.

service runtime
  -> client credential
  -> client_id
  -> allowed scopes/audiences
  -> short-lived access token
  -> resource server policy
  -> audit event

Can the service get a token?

Can we prove exactly which service can do exactly which action, against which resource, under which environment/tenant, and can we revoke/observe it during failure or compromise?