toph/ci-actions
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

refactor: migrate from S3 to Attic binary cache

Browse source 
Replace low-level S3 operations with native Attic client for better
performance, simplicity, and proper Nix binary cache protocol support.

Changes:
- Replace 'nix copy' + S3 with 'attic push'
- Remove S3_ACCESS_KEY, S3_SECRET_KEY, NIX_SIGNING_KEY requirements
- Add ATTIC_TOKEN requirement (explicit per-repo security)
- Default to 'ci' cache instead of 'toph'
- Update Nomad fetch task to pull from Attic instead of S3
- Simplify push-nix-cache to single attic push command
- Update documentation with new security model

Security:
- ATTIC_TOKEN must be explicitly provided as Forgejo secret
- Prevents untrusted repos from pushing to cache
- Separate ci/toph caches for different trust levels

Benefits:
- Simpler: Single command instead of sign + copy + sync
- Faster: Native Attic protocol vs S3 object storage
- Safer: Explicit opt-in prevents unauthorized cache writes
- Standards-compliant: Proper Nix binary cache protocol

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Christopher Mühl 2026-02-27 21:19:09 +01:00
parent 694c591332
commit 19468d38d8
No known key found for this signature in database
GPG key ID: 925AC7D69955293F
5 changed files with 93 additions and 113 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

93
README.md
View file

@ -6,11 +6,11 @@ Reusable Forgejo/Gitea actions for toph's infrastructure.
### `deploy-nix-site` (Recommended) ### `deploy-nix-site` (Recommended)
Deploy a static site built with Nix flake to S3 and Nomad. Provides proper isolation and reproducibility. Deploy a static site built with Nix flake to Attic binary cache and Nomad. Provides proper isolation and reproducibility.
**Requirements:** **Requirements:**
- Repository must have a `flake.nix` with a default package output - Repository must have a `flake.nix` with a default package output
- Runner label: `nix` (uses `docker://nixos/nix:latest`) - Runner label: `nix` (uses `docker://registry.toph.so/nix-runner:latest`)
**Usage:** **Usage:**
@ -31,9 +31,6 @@ jobs:
site-name: mysite site-name: mysite
traefik-rule: Host(`mysite.example.com`) traefik-rule: Host(`mysite.example.com`)
env: env:
S3_ACCESS_KEY: ${{ secrets.S3_ACCESS_KEY }}
S3_SECRET_KEY: ${{ secrets.S3_SECRET_KEY }}
NIX_SIGNING_KEY: ${{ secrets.NIX_SIGNING_KEY }}
NOMAD_TOKEN: ${{ secrets.NOMAD_TOKEN }} NOMAD_TOKEN: ${{ secrets.NOMAD_TOKEN }}
``` ```
@ -41,7 +38,17 @@ jobs:
- `site-name` (required): Site identifier - `site-name` (required): Site identifier
- `traefik-rule` (required): Traefik routing rule - `traefik-rule` (required): Traefik routing rule
- `flake-output` (optional): Nix flake output, defaults to `.#` - `flake-output` (optional): Nix flake output, defaults to `.#`
- `s3-endpoint` (optional): S3 endpoint, defaults to `https://s3.toph.so` - `cache-name` (optional): Attic cache name, defaults to `ci`
- `s3-endpoint` (optional): S3 endpoint (for deployment artifacts), defaults to `https://s3.toph.so`
**Environment variables:**
- `ATTIC_TOKEN`: Attic CI token for pushing to cache (set via Forgejo secrets)
- `NOMAD_TOKEN`: Nomad ACL token for the `static-sites` namespace
**Notes:**
- Build artifacts are automatically pushed to the Attic binary cache at `cache.toph.so/ci`
- Uses the `ci` cache by default (lower trust, shorter retention)
- Pass `cache-name: toph` to use the main cache for trusted repos
**Example flake.nix:** **Example flake.nix:**
@ -175,67 +182,43 @@ cd /srv/infra
nix run .#bosun -- run s3 nix run .#bosun -- run s3
``` ```
### 2. Generate binary cache signing keys ### 2. Bootstrap Attic and create CI cache
Deploy Attic and run the bootstrap script:
```bash ```bash
nix-store --generate-binary-cache-key cache.toph.so cache-priv-key.pem cache-pub-key.pem cd /srv/infra
nix run .#bosun -- run attic
# Public key (add to nix.conf trusted-public-keys on hosts that will fetch): nix run .#attic-bootstrap
cat cache-pub-key.pem
# Example: cache.toph.so:9zFo64TPnxaQeyFM6NS9ou2Fd8OQv4Ia+MuLMjLBYjY=
# Private key (store in Forgejo secrets):
cat cache-priv-key.pem
# Keep this secret!
``` ```
### 3. Create S3 buckets This creates:
- `toph` cache: High-trust cache for manual/trusted builds (priority 40, 180d retention)
- `ci` cache: Lower-trust cache for CI builds (priority 30, 90d retention)
### 3. Get the CI token
```bash ```bash
# Configure AWS CLI nomad var get nomad/jobs/attic/tokens | jq -r '.ci'
export AWS_ACCESS_KEY_ID=<your-access-key>
export AWS_SECRET_ACCESS_KEY=<your-secret-key>
export AWS_ENDPOINT_URL=https://s3.toph.so
export AWS_EC2_METADATA_DISABLED=true
# Create binary cache bucket
aws s3 mb s3://nix-cache
# (Optional) Create artifacts bucket for non-Nix deployments
aws s3 mb s3://artifacts
aws s3api put-bucket-acl --bucket artifacts --acl public-read
``` ```
### 4. Configure alvin to trust the binary cache ### 4. Add Forgejo secrets
Add to `/srv/infra/hosts/alvin/default.nix`: **Security Model:**
- The `ATTIC_TOKEN` must be explicitly provided as a Forgejo secret
- This prevents untrusted repositories from pushing arbitrary binaries to your cache
- Only repositories with the secret configured can push to the cache
```nix **Per-repository secrets** (Settings → Secrets):
nix.settings = { - `ATTIC_TOKEN`: CI token from step 3 (for `ci` cache access)
substituters = [ - `NOMAD_TOKEN`: Auto-synced by `nomad-acl-forgejo-sync` on alvin
"https://cache.nixos.org"
"s3://nix-cache?endpoint=s3.toph.so&scheme=https"
];
trusted-public-keys = [
"cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
"cache.toph.so:9zFo64TPnxaQeyFM6NS9ou2Fd8OQv4Ia+MuLMjLBYjY=" # Your public key
];
};
```
Then deploy: `sudo nixos-rebuild switch --flake .#alvin` **Organization-wide secrets** (for trusted internal repos):
Set these at the organization level to avoid configuring each repo individually.
### 5. Add Forgejo secrets **Cache selection:**
- Use `cache-name: ci` (default) for untrusted/external repos → 90-day retention
In your repository settings (or organization settings for global secrets): - Use `cache-name: toph` for trusted repos → 180-day retention, requires root token
- `S3_ACCESS_KEY`: S3 access key
- `S3_SECRET_KEY`: S3 secret key
- `NIX_SIGNING_KEY`: Contents of `cache-priv-key.pem`
- `NOMAD_TOKEN`: Auto-synced by `nomad-acl-forgejo-sync` on alvin (or set manually from `cat /var/lib/nomad-acl/ci.token`)
### 6. Configure SSH access from runner to alvin
The runner needs to pull store paths to alvin's `/nix/store`. Add the runner's SSH key to alvin or use an agent socket mount.
## Examples ## Examples

56
deploy-nix-site/action.yaml
View file

@ -20,15 +20,17 @@ inputs:
required: false required: false
default: 'https://s3.toph.so' default: 'https://s3.toph.so'
cache-name:
description: 'Attic cache name'
required: false
default: 'ci'
runs: runs:
using: composite using: composite
steps: steps:
- name: Install tools - name: Set environment
shell: bash shell: bash
run: | run: |
# Install AWS CLI
nix profile install nixpkgs#awscli2
# Set Nomad connection # Set Nomad connection
echo "NOMAD_ADDR=http://alvin:4646" >> $GITHUB_ENV echo "NOMAD_ADDR=http://alvin:4646" >> $GITHUB_ENV
echo "NOMAD_TOKEN=${{ env.NOMAD_TOKEN }}" >> $GITHUB_ENV echo "NOMAD_TOKEN=${{ env.NOMAD_TOKEN }}" >> $GITHUB_ENV
@ -36,15 +38,8 @@ runs:
- name: Build site with Nix - name: Build site with Nix
shell: bash shell: bash
run: | run: |
# Configure S3 as substituter to pull cached dependencies # Build with Attic cache as substituter (pre-configured in nix.conf)
export AWS_ACCESS_KEY_ID="${{ env.S3_ACCESS_KEY }}" nix build ${{ inputs.flake-output }} --print-build-logs
export AWS_SECRET_ACCESS_KEY="${{ env.S3_SECRET_KEY }}"
# Build with S3 cache as substituter (fetches cached deps)
nix build ${{ inputs.flake-output }} \
--print-build-logs \
--option substituters "https://cache.nixos.org s3://nix-cache?endpoint=${{ inputs.s3-endpoint }}&scheme=https" \
--option trusted-public-keys "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= $(cat /tmp/cache-pub-key.pem 2>/dev/null || echo '')"
# Get the store path # Get the store path
STORE_PATH=$(readlink -f result) STORE_PATH=$(readlink -f result)
@ -54,27 +49,22 @@ runs:
echo "STORE_HASH=$STORE_HASH" >> $GITHUB_ENV echo "STORE_HASH=$STORE_HASH" >> $GITHUB_ENV
echo "📦 Built: $STORE_PATH" echo "📦 Built: $STORE_PATH"
- name: Push to binary cache - name: Push to Attic cache
shell: bash shell: bash
env:
ATTIC_TOKEN: ${{ env.ATTIC_TOKEN }}
run: | run: |
# Configure S3 binary cache # Configure attic client with explicit token (not relying on mounted config)
export AWS_ACCESS_KEY_ID="${{ env.S3_ACCESS_KEY }}" mkdir -p ~/.config/attic
export AWS_SECRET_ACCESS_KEY="${{ env.S3_SECRET_KEY }}" cat > ~/.config/attic/config.toml <<EOF
[servers.cache]
endpoint = "https://cache.toph.so"
token = "${ATTIC_TOKEN}"
EOF
# Write signing key to temporary file # Push entire closure to Attic cache
echo "${{ env.NIX_SIGNING_KEY }}" > /tmp/nix-signing-key.pem attic push "${{ inputs.cache-name }}" "$STORE_PATH"
chmod 600 /tmp/nix-signing-key.pem echo "✅ Pushed to binary cache: $STORE_HASH"
# Push entire closure (derivation + all dependencies) to cache
nix copy \
--to "s3://nix-cache?endpoint=${{ inputs.s3-endpoint }}&scheme=https&secret-key=/tmp/nix-signing-key.pem" \
--derivation \
"$STORE_PATH"
# Clean up key file
rm -f /tmp/nix-signing-key.pem
echo "✅ Pushed to binary cache: $STORE_HASH (with all dependencies)"
- name: Deploy via Nomad - name: Deploy via Nomad
shell: bash shell: bash
@ -133,12 +123,10 @@ runs:
"command": "/bin/sh", "command": "/bin/sh",
"args": [ "args": [
"-c", "-c",
"nix copy --from 's3://nix-cache?endpoint=${{ inputs.s3-endpoint }}&scheme=https' '${STORE_PATH}' && cp -r ${STORE_PATH}/* /alloc/data/" "nix copy --from 'https://cache.toph.so/${{ inputs.cache-name }}' '${STORE_PATH}' && cp -r ${STORE_PATH}/* /alloc/data/"
] ]
}, },
"Env": { "Env": {
"AWS_ACCESS_KEY_ID": "${{ env.S3_ACCESS_KEY }}",
"AWS_SECRET_ACCESS_KEY": "${{ env.S3_SECRET_KEY }}",
"STORE_PATH": "${{ env.STORE_PATH }}" "STORE_PATH": "${{ env.STORE_PATH }}"
}, },
"VolumeMounts": [{ "VolumeMounts": [{

18
deploy-static-site/action.yaml
View file

@ -1,9 +1,8 @@
name: Deploy Static Site name: Deploy Static Site
description: Build site with Nix, push tarball to S3, deploy via Nomad with shared static-server image description: Build site with Nix, push tarball to S3, deploy via Nomad with shared static-server image
# Required env vars: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, NOMAD_TOKEN # Required env vars: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, NOMAD_TOKEN, ATTIC_TOKEN
# NOMAD_ADDR is injected by the Forgejo runner via container.options # NOMAD_ADDR is injected by the Forgejo runner via container.options
# Optional env vars: NIX_SIGNING_KEY (if set, signs and pushes Nix closure to S3 binary cache)
inputs: inputs:
domain: domain:
@ -31,10 +30,15 @@ inputs:
default: 'https://s3.toph.so' default: 'https://s3.toph.so'
s3-bucket: s3-bucket:
description: 'S3 bucket for site tarballs and Nix cache' description: 'S3 bucket for site tarballs'
required: false required: false
default: 'nix-cache' default: 'nix-cache'
cache-name:
description: 'Attic cache name'
required: false
default: 'ci'
smoke-test: smoke-test:
description: 'Run a smoke test against the domain after deploy' description: 'Run a smoke test against the domain after deploy'
required: false required: false
@ -47,13 +51,13 @@ runs:
shell: bash shell: bash
run: nix build ".#${{ inputs.flake-output }}" --out-link result-site run: nix build ".#${{ inputs.flake-output }}" --out-link result-site
- name: Sign and push Nix closure to S3 cache - name: Push Nix closure to Attic cache
if: env.NIX_SIGNING_KEY != ''
uses: https://git.toph.so/toph/ci-actions/push-nix-cache@main uses: https://git.toph.so/toph/ci-actions/push-nix-cache@main
with: with:
store-path: ./result-site store-path: ./result-site
s3-endpoint: ${{ inputs.s3-endpoint }} cache-name: ${{ inputs.cache-name }}
s3-bucket: ${{ inputs.s3-bucket }} env:
ATTIC_TOKEN: ${{ env.ATTIC_TOKEN }}
- name: Upload site tarball to S3 - name: Upload site tarball to S3
shell: bash shell: bash

2
deploy-static-site/nomad-job.nix
View file

@ -51,7 +51,7 @@ let
"traefik.http.routers.${jobId}.tls.certresolver=letsencrypt" "traefik.http.routers.${jobId}.tls.certresolver=letsencrypt"
]; ];
Checks = [ Checks = [
{ Type = "http"; Path = "/"; Interval = 30000000000; Timeout = 5000000000; } { Type = "http"; Path = "/"; Interval = 5000000000; Timeout = 5000000000; }
]; ];
} }
]; ];

37
push-nix-cache/action.yaml
View file

@ -1,33 +1,38 @@
name: Push Nix Cache name: Push Nix Cache
description: Sign a Nix store path and push it to the S3 binary cache description: Push a Nix store path to the Attic binary cache
# Required env vars: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, NIX_SIGNING_KEY # Required env var: ATTIC_TOKEN
inputs: inputs:
store-path: store-path:
description: 'Path to the Nix store symlink or derivation to push (e.g. ./result)' description: 'Path to the Nix store symlink or derivation to push (e.g. ./result)'
required: true required: true
s3-endpoint: cache-name:
description: 'S3 endpoint URL' description: 'Attic cache name'
required: false required: false
default: 'https://s3.toph.so' default: 'ci'
s3-bucket: attic-endpoint:
description: 'S3 bucket used as the Nix binary cache' description: 'Attic server endpoint'
required: false required: false
default: 'nix-cache' default: 'https://cache.toph.so'
runs: runs:
using: composite using: composite
steps: steps:
- name: Sign and push Nix closure - name: Push to Attic cache
shell: bash shell: bash
env:
ATTIC_TOKEN: ${{ env.ATTIC_TOKEN }}
run: | run: |
echo "${NIX_SIGNING_KEY}" > /tmp/nix-key # Configure attic client with explicit token (not relying on mounted config)
nix store sign -k /tmp/nix-key --recursive "${{ inputs.store-path }}" mkdir -p ~/.config/attic
rm /tmp/nix-key cat > ~/.config/attic/config.toml <<EOF
nix copy --to "file:///tmp/nix-cache" "${{ inputs.store-path }}" [servers.cache]
aws s3 sync /tmp/nix-cache \ endpoint = "${{ inputs.attic-endpoint }}"
"s3://${{ inputs.s3-bucket }}" \ token = "${ATTIC_TOKEN}"
--endpoint-url "${{ inputs.s3-endpoint }}" EOF
# attic push automatically signs and uploads the entire closure
attic push "${{ inputs.cache-name }}" "${{ inputs.store-path }}"