refactor: migrate from S3 to Attic binary cache

Replace low-level S3 operations with native Attic client for better performance, simplicity, and proper Nix binary cache protocol support. Changes: - Replace 'nix copy' + S3 with 'attic push' - Remove S3_ACCESS_KEY, S3_SECRET_KEY, NIX_SIGNING_KEY requirements - Add ATTIC_TOKEN requirement (explicit per-repo security) - Default to 'ci' cache instead of 'toph' - Update Nomad fetch task to pull from Attic instead of S3 - Simplify push-nix-cache to single attic push command - Update documentation with new security model Security: - ATTIC_TOKEN must be explicitly provided as Forgejo secret - Prevents untrusted repos from pushing to cache - Separate ci/toph caches for different trust levels Benefits: - Simpler: Single command instead of sign + copy + sync - Faster: Native Attic protocol vs S3 object storage - Safer: Explicit opt-in prevents unauthorized cache writes - Standards-compliant: Proper Nix binary cache protocol Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-02-27 21:19:09 +01:00 · 2026-02-27 21:19:09 +01:00 · 19468d38d8
commit 19468d38d8
parent 694c591332
5 changed files with 93 additions and 113 deletions
--- a/README.md
+++ b/README.md
@ -6,11 +6,11 @@ Reusable Forgejo/Gitea actions for toph's infrastructure.

 ### `deploy-nix-site` (Recommended)

-Deploy a static site built with Nix flake to S3 and Nomad. Provides proper isolation and reproducibility.
+Deploy a static site built with Nix flake to Attic binary cache and Nomad. Provides proper isolation and reproducibility.

 **Requirements:**
 - Repository must have a `flake.nix` with a default package output
- Runner label: `nix` (uses `docker://nixos/nix:latest`)
+- Runner label: `nix` (uses `docker://registry.toph.so/nix-runner:latest`)

 **Usage:**

@ -31,9 +31,6 @@ jobs:
          site-name: mysite
          traefik-rule: Host(`mysite.example.com`)
        env:
-          S3_ACCESS_KEY: ${{ secrets.S3_ACCESS_KEY }}
-          S3_SECRET_KEY: ${{ secrets.S3_SECRET_KEY }}
-          NIX_SIGNING_KEY: ${{ secrets.NIX_SIGNING_KEY }}
          NOMAD_TOKEN: ${{ secrets.NOMAD_TOKEN }}
 ```

@ -41,7 +38,17 @@ jobs:
 - `site-name` (required): Site identifier
 - `traefik-rule` (required): Traefik routing rule
 - `flake-output` (optional): Nix flake output, defaults to `.#`
- `s3-endpoint` (optional): S3 endpoint, defaults to `https://s3.toph.so`
+- `cache-name` (optional): Attic cache name, defaults to `ci`
+- `s3-endpoint` (optional): S3 endpoint (for deployment artifacts), defaults to `https://s3.toph.so`
+
+**Environment variables:**
+- `ATTIC_TOKEN`: Attic CI token for pushing to cache (set via Forgejo secrets)
+- `NOMAD_TOKEN`: Nomad ACL token for the `static-sites` namespace
+
+**Notes:**
+- Build artifacts are automatically pushed to the Attic binary cache at `cache.toph.so/ci`
+- Uses the `ci` cache by default (lower trust, shorter retention)
+- Pass `cache-name: toph` to use the main cache for trusted repos

 **Example flake.nix:**

@ -175,67 +182,43 @@ cd /srv/infra
 nix run .#bosun -- run s3
 ```

-### 2. Generate binary cache signing keys
+### 2. Bootstrap Attic and create CI cache
+
+Deploy Attic and run the bootstrap script:

 ```bash
-nix-store --generate-binary-cache-key cache.toph.so cache-priv-key.pem cache-pub-key.pem
-
-# Public key (add to nix.conf trusted-public-keys on hosts that will fetch):
-cat cache-pub-key.pem
-# Example: cache.toph.so:9zFo64TPnxaQeyFM6NS9ou2Fd8OQv4Ia+MuLMjLBYjY=
-
-# Private key (store in Forgejo secrets):
-cat cache-priv-key.pem
-# Keep this secret!
+cd /srv/infra
+nix run .#bosun -- run attic
+nix run .#attic-bootstrap
 ```

-### 3. Create S3 buckets
+This creates:
+- `toph` cache: High-trust cache for manual/trusted builds (priority 40, 180d retention)
+- `ci` cache: Lower-trust cache for CI builds (priority 30, 90d retention)
+
+### 3. Get the CI token

 ```bash
-# Configure AWS CLI
-export AWS_ACCESS_KEY_ID=<your-access-key>
-export AWS_SECRET_ACCESS_KEY=<your-secret-key>
-export AWS_ENDPOINT_URL=https://s3.toph.so
-export AWS_EC2_METADATA_DISABLED=true
-
-# Create binary cache bucket
-aws s3 mb s3://nix-cache
-
-# (Optional) Create artifacts bucket for non-Nix deployments
-aws s3 mb s3://artifacts
-aws s3api put-bucket-acl --bucket artifacts --acl public-read
+nomad var get nomad/jobs/attic/tokens | jq -r '.ci'
 ```

-### 4. Configure alvin to trust the binary cache
+### 4. Add Forgejo secrets

-Add to `/srv/infra/hosts/alvin/default.nix`:
+**Security Model:**
+- The `ATTIC_TOKEN` must be explicitly provided as a Forgejo secret
+- This prevents untrusted repositories from pushing arbitrary binaries to your cache
+- Only repositories with the secret configured can push to the cache

-```nix
-nix.settings = {
-  substituters = [
-    "https://cache.nixos.org"
-    "s3://nix-cache?endpoint=s3.toph.so&scheme=https"
-  ];
-  trusted-public-keys = [
-    "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
-    "cache.toph.so:9zFo64TPnxaQeyFM6NS9ou2Fd8OQv4Ia+MuLMjLBYjY="  # Your public key
-  ];
-};
-```
+**Per-repository secrets** (Settings → Secrets):
+- `ATTIC_TOKEN`: CI token from step 3 (for `ci` cache access)
+- `NOMAD_TOKEN`: Auto-synced by `nomad-acl-forgejo-sync` on alvin

-Then deploy: `sudo nixos-rebuild switch --flake .#alvin`
+**Organization-wide secrets** (for trusted internal repos):
+Set these at the organization level to avoid configuring each repo individually.

-### 5. Add Forgejo secrets
-
-In your repository settings (or organization settings for global secrets):
- `S3_ACCESS_KEY`: S3 access key
- `S3_SECRET_KEY`: S3 secret key
- `NIX_SIGNING_KEY`: Contents of `cache-priv-key.pem`
- `NOMAD_TOKEN`: Auto-synced by `nomad-acl-forgejo-sync` on alvin (or set manually from `cat /var/lib/nomad-acl/ci.token`)
-
-### 6. Configure SSH access from runner to alvin
-
-The runner needs to pull store paths to alvin's `/nix/store`. Add the runner's SSH key to alvin or use an agent socket mount.
+**Cache selection:**
+- Use `cache-name: ci` (default) for untrusted/external repos → 90-day retention
+- Use `cache-name: toph` for trusted repos → 180-day retention, requires root token

 ## Examples

--- a/deploy-nix-site/action.yaml
+++ b/deploy-nix-site/action.yaml
@ -20,15 +20,17 @@ inputs:
    required: false
    default: 'https://s3.toph.so'

+  cache-name:
+    description: 'Attic cache name'
+    required: false
+    default: 'ci'
+
 runs:
  using: composite
  steps:
-    - name: Install tools
+    - name: Set environment
      shell: bash
      run: |
-        # Install AWS CLI
-        nix profile install nixpkgs#awscli2
-
        # Set Nomad connection
        echo "NOMAD_ADDR=http://alvin:4646" >> $GITHUB_ENV
        echo "NOMAD_TOKEN=${{ env.NOMAD_TOKEN }}" >> $GITHUB_ENV
@ -36,15 +38,8 @@ runs:
    - name: Build site with Nix
      shell: bash
      run: |
-        # Configure S3 as substituter to pull cached dependencies
-        export AWS_ACCESS_KEY_ID="${{ env.S3_ACCESS_KEY }}"
-        export AWS_SECRET_ACCESS_KEY="${{ env.S3_SECRET_KEY }}"
-
-        # Build with S3 cache as substituter (fetches cached deps)
-        nix build ${{ inputs.flake-output }} \
-          --print-build-logs \
-          --option substituters "https://cache.nixos.org s3://nix-cache?endpoint=${{ inputs.s3-endpoint }}&scheme=https" \
-          --option trusted-public-keys "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= $(cat /tmp/cache-pub-key.pem 2>/dev/null || echo '')"
+        # Build with Attic cache as substituter (pre-configured in nix.conf)
+        nix build ${{ inputs.flake-output }} --print-build-logs

        # Get the store path
        STORE_PATH=$(readlink -f result)
@ -54,27 +49,22 @@ runs:
        echo "STORE_HASH=$STORE_HASH" >> $GITHUB_ENV
        echo "📦 Built: $STORE_PATH"

-    - name: Push to binary cache
+    - name: Push to Attic cache
      shell: bash
+      env:
+        ATTIC_TOKEN: ${{ env.ATTIC_TOKEN }}
      run: |
-        # Configure S3 binary cache
-        export AWS_ACCESS_KEY_ID="${{ env.S3_ACCESS_KEY }}"
-        export AWS_SECRET_ACCESS_KEY="${{ env.S3_SECRET_KEY }}"
+        # Configure attic client with explicit token (not relying on mounted config)
+        mkdir -p ~/.config/attic
+        cat > ~/.config/attic/config.toml <<EOF
+        [servers.cache]
+        endpoint = "https://cache.toph.so"
+        token = "${ATTIC_TOKEN}"
+        EOF

-        # Write signing key to temporary file
-        echo "${{ env.NIX_SIGNING_KEY }}" > /tmp/nix-signing-key.pem
-        chmod 600 /tmp/nix-signing-key.pem
-
-        # Push entire closure (derivation + all dependencies) to cache
-        nix copy \
-          --to "s3://nix-cache?endpoint=${{ inputs.s3-endpoint }}&scheme=https&secret-key=/tmp/nix-signing-key.pem" \
-          --derivation \
-          "$STORE_PATH"
-
-        # Clean up key file
-        rm -f /tmp/nix-signing-key.pem
-
-        echo "✅ Pushed to binary cache: $STORE_HASH (with all dependencies)"
+        # Push entire closure to Attic cache
+        attic push "${{ inputs.cache-name }}" "$STORE_PATH"
+        echo "✅ Pushed to binary cache: $STORE_HASH"

    - name: Deploy via Nomad
      shell: bash
@ -133,12 +123,10 @@ runs:
                    "command": "/bin/sh",
                    "args": [
                      "-c",
-                      "nix copy --from 's3://nix-cache?endpoint=${{ inputs.s3-endpoint }}&scheme=https' '${STORE_PATH}' && cp -r ${STORE_PATH}/* /alloc/data/"
+                      "nix copy --from 'https://cache.toph.so/${{ inputs.cache-name }}' '${STORE_PATH}' && cp -r ${STORE_PATH}/* /alloc/data/"
                    ]
                  },
                  "Env": {
-                    "AWS_ACCESS_KEY_ID": "${{ env.S3_ACCESS_KEY }}",
-                    "AWS_SECRET_ACCESS_KEY": "${{ env.S3_SECRET_KEY }}",
                    "STORE_PATH": "${{ env.STORE_PATH }}"
                  },
                  "VolumeMounts": [{
--- a/deploy-static-site/action.yaml
+++ b/deploy-static-site/action.yaml
@ -1,9 +1,8 @@
 name: Deploy Static Site
 description: Build site with Nix, push tarball to S3, deploy via Nomad with shared static-server image

-# Required env vars: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, NOMAD_TOKEN
+# Required env vars: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, NOMAD_TOKEN, ATTIC_TOKEN
 # NOMAD_ADDR is injected by the Forgejo runner via container.options
-# Optional env vars: NIX_SIGNING_KEY (if set, signs and pushes Nix closure to S3 binary cache)

 inputs:
  domain:
@ -31,10 +30,15 @@ inputs:
    default: 'https://s3.toph.so'

  s3-bucket:
-    description: 'S3 bucket for site tarballs and Nix cache'
+    description: 'S3 bucket for site tarballs'
    required: false
    default: 'nix-cache'

+  cache-name:
+    description: 'Attic cache name'
+    required: false
+    default: 'ci'
+
  smoke-test:
    description: 'Run a smoke test against the domain after deploy'
    required: false
@ -47,13 +51,13 @@ runs:
      shell: bash
      run: nix build ".#${{ inputs.flake-output }}" --out-link result-site

-    - name: Sign and push Nix closure to S3 cache
-      if: env.NIX_SIGNING_KEY != ''
+    - name: Push Nix closure to Attic cache
      uses: https://git.toph.so/toph/ci-actions/push-nix-cache@main
      with:
        store-path: ./result-site
-        s3-endpoint: ${{ inputs.s3-endpoint }}
-        s3-bucket: ${{ inputs.s3-bucket }}
+        cache-name: ${{ inputs.cache-name }}
+      env:
+        ATTIC_TOKEN: ${{ env.ATTIC_TOKEN }}

    - name: Upload site tarball to S3
      shell: bash
--- a/deploy-static-site/nomad-job.nix
+++ b/deploy-static-site/nomad-job.nix
@ -51,7 +51,7 @@ let
                "traefik.http.routers.${jobId}.tls.certresolver=letsencrypt"
              ];
              Checks = [
-                { Type = "http"; Path = "/"; Interval = 30000000000; Timeout = 5000000000; }
+                { Type = "http"; Path = "/"; Interval = 5000000000; Timeout = 5000000000; }
              ];
            }
          ];
--- a/push-nix-cache/action.yaml
+++ b/push-nix-cache/action.yaml
@ -1,33 +1,38 @@
 name: Push Nix Cache
-description: Sign a Nix store path and push it to the S3 binary cache
+description: Push a Nix store path to the Attic binary cache

-# Required env vars: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, NIX_SIGNING_KEY
+# Required env var: ATTIC_TOKEN

 inputs:
  store-path:
    description: 'Path to the Nix store symlink or derivation to push (e.g. ./result)'
    required: true

-  s3-endpoint:
-    description: 'S3 endpoint URL'
+  cache-name:
+    description: 'Attic cache name'
    required: false
-    default: 'https://s3.toph.so'
+    default: 'ci'

-  s3-bucket:
-    description: 'S3 bucket used as the Nix binary cache'
+  attic-endpoint:
+    description: 'Attic server endpoint'
    required: false
-    default: 'nix-cache'
+    default: 'https://cache.toph.so'

 runs:
  using: composite
  steps:
-    - name: Sign and push Nix closure
+    - name: Push to Attic cache
      shell: bash
+      env:
+        ATTIC_TOKEN: ${{ env.ATTIC_TOKEN }}
      run: |
-        echo "${NIX_SIGNING_KEY}" > /tmp/nix-key
-        nix store sign -k /tmp/nix-key --recursive "${{ inputs.store-path }}"
-        rm /tmp/nix-key
-        nix copy --to "file:///tmp/nix-cache" "${{ inputs.store-path }}"
-        aws s3 sync /tmp/nix-cache \
-          "s3://${{ inputs.s3-bucket }}" \
-          --endpoint-url "${{ inputs.s3-endpoint }}"
+        # Configure attic client with explicit token (not relying on mounted config)
+        mkdir -p ~/.config/attic
+        cat > ~/.config/attic/config.toml <<EOF
+        [servers.cache]
+        endpoint = "${{ inputs.attic-endpoint }}"
+        token = "${ATTIC_TOKEN}"
+        EOF
+
+        # attic push automatically signs and uploads the entire closure
+        attic push "${{ inputs.cache-name }}" "${{ inputs.store-path }}"