Case Study: Secret Leak and History Rewrite

Secret leak adalah salah satu insiden Git paling berbahaya karena banyak engineer salah memahami masalahnya.

Mereka berpikir:

“Saya sudah delete file itu dan commit baru. Berarti secret hilang.”

Salah.

Jika secret pernah masuk commit yang sudah dipush, secret itu masih ada di history, mungkin sudah berada di:

• local clone developer,
• fork,
• CI cache,
• PR diff,
• release artifact,
• logs,
• package/source archive,
• search index,
• backup,
• atau downstream mirror.

Karena itu prinsip insiden secret leak adalah:

Rotate/revoke first. Rewrite history second. Coordinate downstream always.

Materi ini adalah case study operasional.

1. Scenario

Tim RegCase Platform menerima alert dari secret scanning:

Secret type: cloud-provider-access-key
Repository: regcase-core
Branch: feature/referral-bulk-import
Commit: a8c4219
Path: services/referral-import/src/main/resources/application-prod.yml
Detected at: 2026-07-07T03:42:11Z

Ternyata branch feature sudah dipush ke remote dan PR sudah dibuka. PR belum merged ke main.

Tiga jam kemudian ditemukan bahwa commit juga sudah terfetch oleh CI dan beberapa reviewer.

Pertanyaan operasional:

1. Apakah credential harus langsung di-rotate?
2. Apakah commit cukup di-revert?
3. Apakah branch perlu dihapus?
4. Apakah history rewrite diperlukan?
5. Apakah main terdampak?
6. Bagaimana membersihkan clone developer?
7. Bagaimana mencegah leak serupa?

2. Important Distinction: Remove Exposure vs Remove Capability

Menghapus secret dari Git history mengurangi exposure.

Rotate/revoke secret menghilangkan capability.

Jika secret sudah pernah terlihat oleh remote server atau clone lain, jangan menganggap secret masih aman.

Wrong order:

rewrite history -> hope nobody used secret -> rotate later

Correct order:

contain access -> rotate/revoke -> preserve evidence -> clean history -> coordinate cleanup -> prevent recurrence

3. Severity Classification

Classify leak before choosing cleanup depth.

For this case:

• private repository,
• branch pushed to remote,
• CI fetched it,
• credential has production write permissions.

Classification: critical, even if not merged to main.

4. Immediate Incident Timeline

The first action is not git filter-repo.

The first action is credential control.

5. Evidence Preservation

Before rewriting history, preserve incident evidence in a secure location.

You do not preserve the secret in normal issue comments or PR descriptions.

Evidence bundle:

secret-incident-2026-07-07/
  summary.md
  affected-refs.txt
  affected-commits.txt
  affected-paths.txt
  exposure-window.txt
  credential-rotation-record.md
  ci-runs.txt
  remote-branches-before.txt
  tags-before.txt
  rewrite-plan.md
  verification-after.txt
  downstream-cleanup-instructions.md

Commands:

git for-each-ref --format='%(refname) %(objectname)' > affected-refs-before.txt

git log --all --format='%H %ad %an %s' --date=iso \
  -- services/referral-import/src/main/resources/application-prod.yml \
  > affected-commits.txt

git grep -n 'AKIA\|SECRET\|PRIVATE_KEY\|password' $(git rev-list --all) \
  > raw-secret-grep-findings.txt

Store evidence according to security policy. Avoid copying secret values into broad-access documents.

6. Determine Rewrite Scope

Ask:

1. Is the secret only in an unmerged feature branch?
2. Did it reach main?
3. Did it reach a release branch/tag?
4. Did it reach public forks or mirrors?
5. Did CI artifact include it?
6. Did dependency consumers vendor this repository?

Decision matrix:

In this scenario: pushed feature branch, CI fetch, no main merge.

Recommended:

• rotate production credential,
• delete or rewrite feature branch,
• invalidate CI caches/artifacts containing secret,
• tell reviewers to discard affected branch,
• add prevention.

7. Why Revert Is Not Enough

Suppose commit a8c4219 adds secret, and commit b91d421 deletes it.

History:

C has no secret in current tree. But B still exists.

Anyone can run:

git show a8c4219:services/referral-import/src/main/resources/application-prod.yml

So revert/delete commit only fixes current snapshot. It does not remove historical exposure.

8. Cleaning a Feature Branch Only

If the branch is unmerged and only branch history is affected, the safest path is often to recreate the branch cleanly.

Option A: Interactive rebase if secret is in one recent commit

git checkout feature/referral-bulk-import

git rebase -i a8c4219^

Mark the bad commit as edit, remove secret, amend:

$EDITOR services/referral-import/src/main/resources/application-prod.yml
git add services/referral-import/src/main/resources/application-prod.yml
git commit --amend

git rebase --continue

Then force push with lease:

git push --force-with-lease origin feature/referral-bulk-import

Good when:

• few commits,
• affected branch only,
• no tags/releases,
• low complexity.

Option B: Recreate clean branch from base

git fetch origin

git checkout -b feature/referral-bulk-import-clean origin/main

git cherry-pick <safe-commit-1> <safe-commit-2>
# skip/recreate bad commit without secret

git push origin feature/referral-bulk-import-clean

Then close old PR and open a new one.

Good when:

• branch small,
• safer than complex rewrite,
• PR review can restart cleanly.

9. Cleaning All Repository History with git filter-repo

If secret reached shared history or multiple refs, use a history rewrite tool.

Modern recommendation commonly uses git filter-repo rather than legacy filter-branch.

High-level flow:

Step 1: Mirror clone

git clone --mirror git@github.com:org/regcase-core.git regcase-core-cleanup.git
cd regcase-core-cleanup.git

A mirror clone includes refs beyond normal branch checkout.

Step 2: Backup

git bundle create ../regcase-core-before-secret-cleanup.bundle --all

git for-each-ref --format='%(refname) %(objectname)' \
  > ../refs-before-secret-cleanup.txt

Step 3: Remove file from all history

If the whole file is secret-bearing:

git filter-repo \
  --path services/referral-import/src/main/resources/application-prod.yml \
  --invert-paths

Step 4: Replace secret text everywhere

If secret appears inside otherwise legitimate files, use replacement:

cat > replacements.txt <<'EOF'
regex:AKIA[0-9A-Z]{16}==>REMOVED_CLOUD_ACCESS_KEY
literal:actual-secret-value==>REMOVED_SECRET_VALUE
EOF

git filter-repo --replace-text replacements.txt

Be careful. Replacement files can themselves contain secrets. Store and destroy them according to policy.

Step 5: Verify

git grep -n 'AKIA\|actual-secret-value' $(git rev-list --all) || true

git fsck --no-reflogs --unreachable

git for-each-ref --format='%(refname) %(objectname)' \
  > ../refs-after-secret-cleanup.txt

Verification must search all reachable history, not only current working tree.

Step 6: Push rewritten refs

git push --force --mirror origin

This is destructive. Use only after coordination and approvals.

For high-risk repositories, do not allow one developer to run this alone.

10. Blast Radius of History Rewrite

A history rewrite changes commit identities.

Old graph:

New graph:

Even commits whose patch looks similar may get new hashes because their parent changed.

Consequences:

• open PRs may break,
• local branches diverge,
• release tags may need recreation or deprecation,
• CI caches may reference old SHAs,
• submodules may point to missing commits,
• downstream mirrors may retain old objects,
• signed commits/tags may no longer verify as previous objects,
• audit records must explain why identity changed.

Rewrite is sometimes necessary, but never cheap.

11. Downstream Cleanup Instructions

After rewrite, tell contributors not to merge old history back.

Recommended for most developers:

# Save uncommitted work first
git status

# Fetch rewritten history
git fetch origin --prune --tags

# Reset local main to new origin/main
git checkout main
git reset --hard origin/main

# Delete affected feature branch if it contained leaked secret
git branch -D feature/referral-bulk-import || true

# Recreate work from clean branch or cherry-pick only safe commits
git checkout -b feature/referral-bulk-import-clean origin/main

For severe/public leaks, requiring a fresh clone is often simpler and safer.

Important warning:

Do not push old local branches without checking them.
They may reintroduce the removed secret objects to the remote.

12. CI and Artifact Cleanup

Repository history is not the only surface.

Check:

Example artifact inspection:

jar tf build/libs/referral-import.jar | grep application-prod.yml || true

docker history --no-trunc registry.example.com/referral-import:<tag>

docker save registry.example.com/referral-import:<tag> -o image.tar
# inspect layers in controlled environment

If the secret reached an artifact, publish a corrected artifact with new identity. Do not silently replace artifact bytes under the same release identity.

13. Public Repository Reality

For public repositories, assume the secret is compromised as soon as it is pushed.

Even if you rewrite quickly:

• bots may have cloned it,
• search systems may have indexed it,
• forks may preserve it,
• pull request refs may retain it,
• external archives may exist.

So public leak remediation is:

1. revoke/rotate,
2. investigate usage,
3. remove history as hygiene,
4. notify as required,
5. improve controls.

History rewrite is not containment. Credential rotation is containment.

14. PR Ref and Closed PR Caveat

Hosting platforms may retain PR refs or cached views separately from normal branches. A rewrite of branch history may not instantly remove every visible trace from every UI/cached place.

Operational response:

• close affected PR,
• delete affected branch,
• contact platform support if sensitive data remains accessible in PR cache,
• rotate credential regardless,
• document residual exposure.

Do not promise stakeholders that Git rewrite guarantees universal erasure.

15. Post-Rewrite Verification

Verification checklist:

# Verify no reachable commit contains pattern
git grep -n 'AKIA\|PRIVATE_KEY\|actual-secret-value' $(git rev-list --all) || true

# Verify current tree clean
git grep -n 'AKIA\|PRIVATE_KEY\|actual-secret-value' HEAD || true

# Inspect refs
git for-each-ref --format='%(refname) %(objectname)'

# Run object/fsck check
git fsck --full

# Confirm old bad commit is no longer reachable
git merge-base --is-ancestor a8c4219 main && echo BAD || echo not-reachable

Also verify remote after push:

git clone --mirror git@github.com:org/regcase-core.git verify-clean.git
cd verify-clean.git

git grep -n 'actual-secret-value' $(git rev-list --all) || true

Do not verify only local rewritten clone.

16. Prevention Controls

.gitignore is not enough

.gitignore prevents accidental tracking of untracked files. It does not protect tracked files, past commits, or copied values.

Still useful:

.env
.env.*
*.pem
*.key
application-prod.yml
secrets/

But this is a weak layer.

Use templates

Commit safe sample config:

application-prod.example.yml

Never commit real prod config.

Pre-commit hook

#!/usr/bin/env bash
set -euo pipefail

patterns='AKIA[0-9A-Z]{16}|-----BEGIN PRIVATE KEY-----|password\s*:'

if git diff --cached -U0 | grep -E "$patterns" >/dev/null; then
  echo "Potential secret detected in staged diff." >&2
  echo "Move secrets to secret manager or environment config." >&2
  exit 1
fi

This catches some mistakes early but is bypassable and can false-positive.

Pre-receive hook or server-side scanner

Server-side enforcement is stronger:

#!/usr/bin/env bash
set -euo pipefail

while read -r old new ref; do
  # skip deletions
  [ "$new" = "0000000000000000000000000000000000000000" ] && continue

  range="$old..$new"
  if [ "$old" = "0000000000000000000000000000000000000000" ]; then
    range="$new"
  fi

  if git grep -n -E 'AKIA[0-9A-Z]{16}|-----BEGIN PRIVATE KEY-----' $(git rev-list "$range") >/tmp/secret-findings; then
    echo "Rejected push: potential secret detected" >&2
    cat /tmp/secret-findings >&2
    exit 1
  fi
done

Real systems need better scanners to reduce false positives.

Secret manager policy

The strongest prevention is architectural:

• application reads secrets from secret manager,
• local dev uses .env.local ignored file,
• production config is injected at deploy time,
• no production secret exists in source repository,
• CI tokens are scoped and short-lived.

17. Case Study Resolution

For this incident:

Step 1: Rotate credential

Security revokes the cloud key and issues new credential with narrower scope.

Step 2: Freeze affected branch

Repository admin temporarily blocks updates to feature/referral-bulk-import.

Step 3: Preserve evidence

Incident record stores affected commit, path, detection timestamp, and rotation evidence.

Step 4: Recreate clean branch

Because PR was unmerged and branch was small:

git checkout -b feature/referral-bulk-import-clean origin/main

git cherry-pick <safe-commit-before-secret>
# manually recreate config change using example config only

git push origin feature/referral-bulk-import-clean

Old branch deleted after coordination:

git push origin --delete feature/referral-bulk-import

Step 5: CI cleanup

Affected CI workspace cache invalidated.

Step 6: PR cleanup

Old PR closed with security-safe note. New PR opened from clean branch.

Step 7: Prevent recurrence

• add secret scanner as required check,
• add .gitignore patterns,
• create application-prod.example.yml,
• remove prod credentials from developer-accessible config,
• require scoped temporary credentials for import testing,
• add pre-commit feedback hook.

18. When Full Rewrite Would Be Required

If the bad commit had reached main, release branch, or tag, cleanup would be stricter.

Full rewrite checklist:

• rotate/revoke all affected credentials,
• identify all affected refs,
• freeze pushes,
• notify repo owners and downstream consumers,
• create mirror clone,
• backup refs/bundle,
• run git filter-repo,
• verify all reachable history clean,
• force-push rewritten refs,
• require fresh clone or hard reset procedure,
• invalidate CI caches/artifacts,
• recreate/deprecate affected release tags,
• record rewrite reason and old/new ref map,
• monitor for old commits being pushed back.

19. Old-to-New Ref Map

For compliance and debugging, preserve a map of old ref tips to new ref tips.

before:
refs/heads/main  91a0d7c...
refs/heads/release/2026.07 0f1a9ee...
refs/tags/v2026.07.2 1bd48cc...

after:
refs/heads/main  c8e12bb...
refs/heads/release/2026.07 d1184af...
refs/tags/v2026.07.2 deprecated/recreated as v2026.07.2-clean

Avoid reusing release tag names silently if consumers may have seen the old tag.

Safer pattern:

• revoke compromised artifact,
• publish v2026.07.2+revoked metadata internally,
• create v2026.07.3 as corrected release,
• explain incident in restricted security record.

20. Communication Template

Internal announcement:

## Git Secret Incident Cleanup Required

A credential was accidentally pushed to `regcase-core` on branch
`feature/referral-bulk-import` at approximately 2026-07-07 03:42 UTC.

The credential has been revoked/rotated. Do not use or copy the old value.

The affected branch has been deleted and replaced by:

`feature/referral-bulk-import-clean`

Required action:

1. Run `git fetch origin --prune`.
2. Delete any local copy of `feature/referral-bulk-import`.
3. Do not push branches based on the old affected history.
4. Recreate local work from `origin/main` or the clean branch.

Contact Security/Platform if you have local commits based on the affected branch.

Do not include the secret value in the announcement.

21. Git Command Decision Table

22. Postmortem Questions

A useful postmortem asks:

• Why did a real production credential exist in developer-accessible files?
• Why did scanner catch it only after push?
• Were credentials overly privileged?
• Did CI expose it further?
• Did branch protection/checks fail to block it?
• Was there a safe sample config path?
• Did docs teach the wrong local setup?
• Did emergency cleanup require tribal knowledge?
• Did downstream developers know how to recover after rewrite?

The answer should improve system design, not blame the person who committed the secret.

23. Mental Model

A Git secret leak has two separate problems:

1. Security capability problem: someone may use the credential.
2. Information persistence problem: Git history may preserve the value.

Credential rotation handles the first.

History rewrite reduces the second.

Neither alone is sufficient.

The strongest engineering response is layered:

Top-tier Git usage is not knowing one magic cleanup command. It is knowing when cleanup is too late to be containment, and designing the system so the same class of incident becomes harder to repeat.

References

• GitHub Docs: Removing sensitive data from a repository.
• GitHub Docs: Secret scanning and credential rotation guidance.
• Git filter-repo project documentation.
• Git documentation: git rebase, git push --force-with-lease, git for-each-ref, git rev-list, git grep, git fsck.
• Git documentation: git tag and release identity considerations.