Reminders

• Final Project Proposals due today!
• Joy Liu guest lecture on 4/22, attendance will be taken

Agenda

1. What Are Secrets?
2. How Secrets Get Leaked
3. Real-World Breaches
4. Defense in Depth
5. Prevention: .gitignore, Pre-Commit Hooks
6. Detection: Secret Scanning
7. Management: CI/CD Secrets, Secrets Managers, Kubernetes
8. Rotation: Automated Rotation, Dynamic Secrets
9. Response: Incident Playbook

What Counts as a Secret?

Any piece of data that grants access to a system or resource:

• API keys: Third-party service access (Stripe, AWS, Twilio)
• Database credentials: Connection strings with username/password
• SSH keys: Server access
• TLS certificates and private keys: HTTPS encryption
• OAuth tokens: User authentication
• Encryption keys: Data protection at rest

Why Do Secrets Matter?

• Secrets are the keys to your infrastructure
• A leaked AWS key can spin up thousands of dollars in compute in minutes
• A leaked database password exposes all your user data
• A leaked API key lets attackers act as your application

Secrets are the easiest attack vector — no need to find a bug in your code if the attacker already has the keys.

Git

# "I'll just hardcode this for now..."
import boto3
client = boto3.client(
    's3',
    aws_access_key_id='AKIAIOSFODNN7EXAMPLE',
    aws_secret_access_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'
)

• Developer hardcodes credentials during development and commits to GitHub
• Even if you delete the file, it's still in git history
• Bots scan public GitHub repos and exploit exposed keys

Secret Sprawl

Secrets tend to multiply and spread across systems:

• .env files checked into repos
• Credentials copy-pasted into Slack messages
• API keys stored in plain-text config files
• Shared passwords in team wikis or Google Docs
• Secrets baked into Docker images

The more places a secret lives, the harder it is to rotate or revoke.

The .env File Trap

# .env
DATABASE_URL=postgres://admin:supersecret@db.example.com:5432/myapp
STRIPE_SECRET_KEY=sk_live_abc123
AWS_ACCESS_KEY_ID=AKIA...

• .env files are convenient for local development, but can get committed to repos accidentally
• They get copied between machines without encryption
• They don't have access controls or audit logs

.env files are not a secrets management solution — they are a starting point.

Environment Variables Aren't Enough

There still are problems:

• Visible in process listings (/proc/<pid>/environ on Linux)
• Inherited by child processes (including ones you didn't intend)
• Logged by accident in crash dumps, debug output, or monitoring tools
• No audit trail: who accessed what, when?
• No rotation mechanism built in

Environment variables are better than hardcoding, but not a complete solution.

Toyota (2022): Secret Key in Public Repo

• A secret key for Toyota's T-Connect service was accidentally committed to a public GitHub repo
• The key remained exposed for nearly 5 years before being discovered
• Potentially exposed email addresses and customer numbers of ~296,000 customers

Lesson: Secrets in git history persist even if the file is later deleted. You have to rotate the key.

CircleCI (2023): Rotate Everything

• CircleCI suffered a security breach and advised all customers to rotate every secret stored in their platform
• Thousands of organizations had to emergency-rotate database passwords, API keys, cloud credentials
• Massive disruption across the industry

Lesson: When a secrets store is compromised, rotation is the only defense. If rotation is painful, you're not prepared.

Uber (2016): Hardcoded AWS Keys

• Engineers stored AWS credentials in a private GitHub repo
• Attackers gained access to the repo and found the keys
• Used the keys to access an S3 bucket with data on 57 million users and drivers
• Uber paid the attackers $100,000 to delete the data and keep quiet
• Uber was later fined $148 million for concealing the breach

Lesson: Private repos are not a security boundary. Secrets don't belong in code, period.

Layers of Secrets Management

No single tool solves secrets management. You need layers:

1. Prevention: Stop secrets from entering code in the first place
2. Detection: Find secrets that slip through
3. Management: Centralize and control access to secrets
4. Rotation: Regularly change secrets to limit exposure
5. Response: Act fast when a leak happens

Prevention: .gitignore

# .gitignore
.env
.env.*
*.pem
*.key
credentials.json
secrets.yaml

• Prevents accidental commits of secret files
• Not sufficient alone — only prevents new commits, not past mistakes
• Must be set up before the first commit containing secrets

Prevention: Pre-Commit Hooks

Local commit hooks can block secrets before they ever reach the remote:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.0
    hooks:
      - id: gitleaks

$ git commit -m "add config"
gitleaks..........................................................Failed
- Finding: AWS Access Key detected

Prevention: CI/CD Platform Secrets

...
    steps:
      - name: Configure AWS
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      - name: Deploy
        run: terraform apply -auto-approve

• Secrets stored encrypted in GitHub's settings, injected at runtime
• Masked in logs — GitHub redacts secret values from output

Prevention: OIDC (No More Long-Lived Keys)

Problem: cloud provider keys need to be long lived

- uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789:role/github-actions
    aws-region: us-east-1

• GitHub proves its identity to AWS using a signed token
• AWS issues temporary credentials (valid for minutes, not forever)
• No long-lived secrets to leak or rotate
• If the token is stolen, it expires quickly

Prevention: CI/CD Best Practices

• Principle of least privilege: Give pipelines only the permissions they need
  • Don't use your personal AWS admin credentials in CI
• Scope secrets appropriately: Use environment-level secrets when possible
  • Separate secrets for staging vs. production
• Never echo secrets in logs
  • Be careful with set -x or verbose mode in shell scripts
• Use short-lived credentials when possible (OIDC, temporary tokens)

Detection: GitHub Secret Scanning

• GitHub automatically scans public repos for known secret patterns (launched Dec 2022)
• Partners with cloud providers (AWS, GCP, Stripe) to auto-revoke leaked keys
• Also available for private repos (GitHub Advanced Security)

This is a safety net, not a strategy — by the time scanning catches it, the secret was already exposed.

Detection: Scanning Git History

Deleting a file doesn't remove it from git history. You need to scan all historical commits.

• gitleaks / truffleHog: Scan repos for secrets across all historical commits
• CI integration: Run secret scanning as a pipeline step
• Catches secrets that .gitignore and pre-commit hooks missed

# Scan entire repo history with gitleaks
gitleaks detect --source . --verbose

Management: Why Use a Secrets Manager?

Environment variables and CI secrets get you started, but at scale you need:

• Centralized storage: One place for all secrets
• Access control: Who/what can read each secret?
• Audit logging: Who accessed what and when?
• Rotation: Automatically change secrets on a schedule
• Dynamic secrets: Generate short-lived credentials on demand
• Encryption: Secrets encrypted at rest and in transit

Management: The Landscape

Management: AWS Secrets Manager

aws secretsmanager create-secret \
  --name prod/database/password \
  --secret-string "my-db-password"
aws secretsmanager get-secret-value \
  --secret-id prod/database/password

• Secrets encrypted with KMS
• Automatic rotation with Lambda functions
• Fine-grained IAM policies control access
• Audit trail via CloudTrail (logging)

Management: HashiCorp Vault

Vault can generate short-lived, unique credentials on demand:

# Request temporary database credentials
$ vault read database/creds/my-role
Key             Value
---             -----
lease_id        database/creds/my-role/abc123
lease_duration  1h
username        v-app-my-role-xyz789
password        A1b2C3d4E5-temp

• Each application instance gets its own credentials
• Credentials expire automatically after the lease

Management: Kubernetes Secrets

Kubernetes has a built-in Secret resource (as we've been using).

• Can be set in manifests or by command line
• Mounted as files or environment variables in pods
• Anyone with kubectl access can decode them
• Secrets injected at pod startup
• Application reads them like normal environment variables

Management: Kubernetes Secrets

spec:
  containers:
  - name: app
    image: myapp:v1
    env:
    - name: DB_USERNAME
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: password

Management: The Problem with K8s Secrets

• Stored unencrypted in etcd by default
• Anyone with kubectl access can read them: kubectl get secret db-credentials -o yaml
• Secrets in YAML manifests end up in git repos (back to square one!)
• No audit trail of who accessed secrets
• No built-in rotation

Kubernetes Secrets are a delivery mechanism, not a security solution.

Management: External Secrets Operator

Solution: Pull secrets from an external secrets manager into Kubernetes.

• Secrets live in AWS Secrets Manager (or Vault, GCP, etc.)
• External Secrets Operator syncs them into Kubernetes Secrets
• Rotation happens in the external store, K8s stays in sync

Management: External Secrets Operator

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: db-credentials
  data:
  - secretKey: password
    remoteRef:
      key: prod/database/password

Management: Comparing Approaches

Rotation: Why Rotate?

• Limits exposure window — if a secret was leaked, how long can an attacker use it?
• Compliance requirements — PCI-DSS, SOC2, etc. mandate regular rotation
• Employee offboarding — revoke access when people leave
• Blast radius — short-lived credentials mean less damage if compromised

The CircleCI breach forced thousands of organizations to emergency-rotate. If rotation is painful, you're not prepared.

Rotation: Making It Painless

How to make rotation a non-event:

• Applications should read secrets at runtime, not at deploy time
• Use a secrets manager with automatic rotation
• Test rotation in staging before production
• Never hardcode secrets — that makes rotation a full redeploy

Dynamic secrets (Vault) take this further: every credential is short-lived and unique, so there's nothing to rotate — secrets just expire.

Response: When a Secret Leaks

It will happen. The question is how fast you respond.

1. Revoke/rotate the secret immediately — don't just delete the commit
2. Assess the blast radius — what could an attacker access with this secret?
3. Check access logs — was the secret used by anyone unexpected?
4. Deploy the new secret to all systems that need it
5. Post-mortem — how did it happen? How do we prevent it next time?

Deleting a commit or force-pushing does NOT help — the secret is already exposed.

Summary

• Secrets are the keys to your infrastructure — treat them accordingly
• Five layers: Prevention, Detection, Management, Rotation, Response
• Start simple (.gitignore, pre-commit hooks) and build up
• Use CI platform secrets and OIDC for pipelines
• Kubernetes Secrets are a delivery mechanism — pair with an external secrets manager
• Plan for leaks: rotation should be easy, not an emergency scramble