diff --git a/README.md b/README.md index 444e893..2598fb1 100644 --- a/README.md +++ b/README.md @@ -6,11 +6,11 @@ Reusable Forgejo/Gitea actions for toph's infrastructure. ### `deploy-nix-site` (Recommended) -Deploy a static site built with Nix flake to S3 and Nomad. Provides proper isolation and reproducibility. +Deploy a static site built with Nix flake to Attic binary cache and Nomad. Provides proper isolation and reproducibility. **Requirements:** - Repository must have a `flake.nix` with a default package output -- Runner label: `nix` (uses `docker://nixos/nix:latest`) +- Runner label: `nix` (uses `docker://registry.toph.so/nix-runner:latest`) **Usage:** @@ -31,9 +31,6 @@ jobs: site-name: mysite traefik-rule: Host(`mysite.example.com`) env: - S3_ACCESS_KEY: ${{ secrets.S3_ACCESS_KEY }} - S3_SECRET_KEY: ${{ secrets.S3_SECRET_KEY }} - NIX_SIGNING_KEY: ${{ secrets.NIX_SIGNING_KEY }} NOMAD_TOKEN: ${{ secrets.NOMAD_TOKEN }} ``` @@ -41,7 +38,17 @@ jobs: - `site-name` (required): Site identifier - `traefik-rule` (required): Traefik routing rule - `flake-output` (optional): Nix flake output, defaults to `.#` -- `s3-endpoint` (optional): S3 endpoint, defaults to `https://s3.toph.so` +- `cache-name` (optional): Attic cache name, defaults to `ci` +- `s3-endpoint` (optional): S3 endpoint (for deployment artifacts), defaults to `https://s3.toph.so` + +**Environment variables:** +- `ATTIC_TOKEN`: Attic CI token for pushing to cache (set via Forgejo secrets) +- `NOMAD_TOKEN`: Nomad ACL token for the `static-sites` namespace + +**Notes:** +- Build artifacts are automatically pushed to the Attic binary cache at `cache.toph.so/ci` +- Uses the `ci` cache by default (lower trust, shorter retention) +- Pass `cache-name: toph` to use the main cache for trusted repos **Example flake.nix:** @@ -175,67 +182,43 @@ cd /srv/infra nix run .#bosun -- run s3 ``` -### 2. Generate binary cache signing keys +### 2. Bootstrap Attic and create CI cache + +Deploy Attic and run the bootstrap script: ```bash -nix-store --generate-binary-cache-key cache.toph.so cache-priv-key.pem cache-pub-key.pem - -# Public key (add to nix.conf trusted-public-keys on hosts that will fetch): -cat cache-pub-key.pem -# Example: cache.toph.so:9zFo64TPnxaQeyFM6NS9ou2Fd8OQv4Ia+MuLMjLBYjY= - -# Private key (store in Forgejo secrets): -cat cache-priv-key.pem -# Keep this secret! +cd /srv/infra +nix run .#bosun -- run attic +nix run .#attic-bootstrap ``` -### 3. Create S3 buckets +This creates: +- `toph` cache: High-trust cache for manual/trusted builds (priority 40, 180d retention) +- `ci` cache: Lower-trust cache for CI builds (priority 30, 90d retention) + +### 3. Get the CI token ```bash -# Configure AWS CLI -export AWS_ACCESS_KEY_ID= -export AWS_SECRET_ACCESS_KEY= -export AWS_ENDPOINT_URL=https://s3.toph.so -export AWS_EC2_METADATA_DISABLED=true - -# Create binary cache bucket -aws s3 mb s3://nix-cache - -# (Optional) Create artifacts bucket for non-Nix deployments -aws s3 mb s3://artifacts -aws s3api put-bucket-acl --bucket artifacts --acl public-read +nomad var get nomad/jobs/attic/tokens | jq -r '.ci' ``` -### 4. Configure alvin to trust the binary cache +### 4. Add Forgejo secrets -Add to `/srv/infra/hosts/alvin/default.nix`: +**Security Model:** +- The `ATTIC_TOKEN` must be explicitly provided as a Forgejo secret +- This prevents untrusted repositories from pushing arbitrary binaries to your cache +- Only repositories with the secret configured can push to the cache -```nix -nix.settings = { - substituters = [ - "https://cache.nixos.org" - "s3://nix-cache?endpoint=s3.toph.so&scheme=https" - ]; - trusted-public-keys = [ - "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=" - "cache.toph.so:9zFo64TPnxaQeyFM6NS9ou2Fd8OQv4Ia+MuLMjLBYjY=" # Your public key - ]; -}; -``` +**Per-repository secrets** (Settings → Secrets): +- `ATTIC_TOKEN`: CI token from step 3 (for `ci` cache access) +- `NOMAD_TOKEN`: Auto-synced by `nomad-acl-forgejo-sync` on alvin -Then deploy: `sudo nixos-rebuild switch --flake .#alvin` +**Organization-wide secrets** (for trusted internal repos): +Set these at the organization level to avoid configuring each repo individually. -### 5. Add Forgejo secrets - -In your repository settings (or organization settings for global secrets): -- `S3_ACCESS_KEY`: S3 access key -- `S3_SECRET_KEY`: S3 secret key -- `NIX_SIGNING_KEY`: Contents of `cache-priv-key.pem` -- `NOMAD_TOKEN`: Auto-synced by `nomad-acl-forgejo-sync` on alvin (or set manually from `cat /var/lib/nomad-acl/ci.token`) - -### 6. Configure SSH access from runner to alvin - -The runner needs to pull store paths to alvin's `/nix/store`. Add the runner's SSH key to alvin or use an agent socket mount. +**Cache selection:** +- Use `cache-name: ci` (default) for untrusted/external repos → 90-day retention +- Use `cache-name: toph` for trusted repos → 180-day retention, requires root token ## Examples diff --git a/deploy-nix-site/action.yaml b/deploy-nix-site/action.yaml index 684878e..a2eef16 100644 --- a/deploy-nix-site/action.yaml +++ b/deploy-nix-site/action.yaml @@ -20,15 +20,17 @@ inputs: required: false default: 'https://s3.toph.so' + cache-name: + description: 'Attic cache name' + required: false + default: 'ci' + runs: using: composite steps: - - name: Install tools + - name: Set environment shell: bash run: | - # Install AWS CLI - nix profile install nixpkgs#awscli2 - # Set Nomad connection echo "NOMAD_ADDR=http://alvin:4646" >> $GITHUB_ENV echo "NOMAD_TOKEN=${{ env.NOMAD_TOKEN }}" >> $GITHUB_ENV @@ -36,15 +38,8 @@ runs: - name: Build site with Nix shell: bash run: | - # Configure S3 as substituter to pull cached dependencies - export AWS_ACCESS_KEY_ID="${{ env.S3_ACCESS_KEY }}" - export AWS_SECRET_ACCESS_KEY="${{ env.S3_SECRET_KEY }}" - - # Build with S3 cache as substituter (fetches cached deps) - nix build ${{ inputs.flake-output }} \ - --print-build-logs \ - --option substituters "https://cache.nixos.org s3://nix-cache?endpoint=${{ inputs.s3-endpoint }}&scheme=https" \ - --option trusted-public-keys "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= $(cat /tmp/cache-pub-key.pem 2>/dev/null || echo '')" + # Build with Attic cache as substituter (pre-configured in nix.conf) + nix build ${{ inputs.flake-output }} --print-build-logs # Get the store path STORE_PATH=$(readlink -f result) @@ -54,27 +49,22 @@ runs: echo "STORE_HASH=$STORE_HASH" >> $GITHUB_ENV echo "📦 Built: $STORE_PATH" - - name: Push to binary cache + - name: Push to Attic cache shell: bash + env: + ATTIC_TOKEN: ${{ env.ATTIC_TOKEN }} run: | - # Configure S3 binary cache - export AWS_ACCESS_KEY_ID="${{ env.S3_ACCESS_KEY }}" - export AWS_SECRET_ACCESS_KEY="${{ env.S3_SECRET_KEY }}" + # Configure attic client with explicit token (not relying on mounted config) + mkdir -p ~/.config/attic + cat > ~/.config/attic/config.toml < /tmp/nix-signing-key.pem - chmod 600 /tmp/nix-signing-key.pem - - # Push entire closure (derivation + all dependencies) to cache - nix copy \ - --to "s3://nix-cache?endpoint=${{ inputs.s3-endpoint }}&scheme=https&secret-key=/tmp/nix-signing-key.pem" \ - --derivation \ - "$STORE_PATH" - - # Clean up key file - rm -f /tmp/nix-signing-key.pem - - echo "✅ Pushed to binary cache: $STORE_HASH (with all dependencies)" + # Push entire closure to Attic cache + attic push "${{ inputs.cache-name }}" "$STORE_PATH" + echo "✅ Pushed to binary cache: $STORE_HASH" - name: Deploy via Nomad shell: bash @@ -133,12 +123,10 @@ runs: "command": "/bin/sh", "args": [ "-c", - "nix copy --from 's3://nix-cache?endpoint=${{ inputs.s3-endpoint }}&scheme=https' '${STORE_PATH}' && cp -r ${STORE_PATH}/* /alloc/data/" + "nix copy --from 'https://cache.toph.so/${{ inputs.cache-name }}' '${STORE_PATH}' && cp -r ${STORE_PATH}/* /alloc/data/" ] }, "Env": { - "AWS_ACCESS_KEY_ID": "${{ env.S3_ACCESS_KEY }}", - "AWS_SECRET_ACCESS_KEY": "${{ env.S3_SECRET_KEY }}", "STORE_PATH": "${{ env.STORE_PATH }}" }, "VolumeMounts": [{ diff --git a/deploy-static-site/action.yaml b/deploy-static-site/action.yaml index fbb0d56..b626b3b 100644 --- a/deploy-static-site/action.yaml +++ b/deploy-static-site/action.yaml @@ -1,9 +1,8 @@ name: Deploy Static Site description: Build site with Nix, push tarball to S3, deploy via Nomad with shared static-server image -# Required env vars: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, NOMAD_TOKEN +# Required env vars: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, NOMAD_TOKEN, ATTIC_TOKEN # NOMAD_ADDR is injected by the Forgejo runner via container.options -# Optional env vars: NIX_SIGNING_KEY (if set, signs and pushes Nix closure to S3 binary cache) inputs: domain: @@ -31,10 +30,15 @@ inputs: default: 'https://s3.toph.so' s3-bucket: - description: 'S3 bucket for site tarballs and Nix cache' + description: 'S3 bucket for site tarballs' required: false default: 'nix-cache' + cache-name: + description: 'Attic cache name' + required: false + default: 'ci' + smoke-test: description: 'Run a smoke test against the domain after deploy' required: false @@ -47,13 +51,13 @@ runs: shell: bash run: nix build ".#${{ inputs.flake-output }}" --out-link result-site - - name: Sign and push Nix closure to S3 cache - if: env.NIX_SIGNING_KEY != '' + - name: Push Nix closure to Attic cache uses: https://git.toph.so/toph/ci-actions/push-nix-cache@main with: store-path: ./result-site - s3-endpoint: ${{ inputs.s3-endpoint }} - s3-bucket: ${{ inputs.s3-bucket }} + cache-name: ${{ inputs.cache-name }} + env: + ATTIC_TOKEN: ${{ env.ATTIC_TOKEN }} - name: Upload site tarball to S3 shell: bash diff --git a/deploy-static-site/nomad-job.nix b/deploy-static-site/nomad-job.nix index d7be3a1..68ff5d9 100644 --- a/deploy-static-site/nomad-job.nix +++ b/deploy-static-site/nomad-job.nix @@ -51,7 +51,7 @@ let "traefik.http.routers.${jobId}.tls.certresolver=letsencrypt" ]; Checks = [ - { Type = "http"; Path = "/"; Interval = 30000000000; Timeout = 5000000000; } + { Type = "http"; Path = "/"; Interval = 5000000000; Timeout = 5000000000; } ]; } ]; diff --git a/push-nix-cache/action.yaml b/push-nix-cache/action.yaml index bc79e91..199b3c4 100644 --- a/push-nix-cache/action.yaml +++ b/push-nix-cache/action.yaml @@ -1,33 +1,38 @@ name: Push Nix Cache -description: Sign a Nix store path and push it to the S3 binary cache +description: Push a Nix store path to the Attic binary cache -# Required env vars: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, NIX_SIGNING_KEY +# Required env var: ATTIC_TOKEN inputs: store-path: description: 'Path to the Nix store symlink or derivation to push (e.g. ./result)' required: true - s3-endpoint: - description: 'S3 endpoint URL' + cache-name: + description: 'Attic cache name' required: false - default: 'https://s3.toph.so' + default: 'ci' - s3-bucket: - description: 'S3 bucket used as the Nix binary cache' + attic-endpoint: + description: 'Attic server endpoint' required: false - default: 'nix-cache' + default: 'https://cache.toph.so' runs: using: composite steps: - - name: Sign and push Nix closure + - name: Push to Attic cache shell: bash + env: + ATTIC_TOKEN: ${{ env.ATTIC_TOKEN }} run: | - echo "${NIX_SIGNING_KEY}" > /tmp/nix-key - nix store sign -k /tmp/nix-key --recursive "${{ inputs.store-path }}" - rm /tmp/nix-key - nix copy --to "file:///tmp/nix-cache" "${{ inputs.store-path }}" - aws s3 sync /tmp/nix-cache \ - "s3://${{ inputs.s3-bucket }}" \ - --endpoint-url "${{ inputs.s3-endpoint }}" + # Configure attic client with explicit token (not relying on mounted config) + mkdir -p ~/.config/attic + cat > ~/.config/attic/config.toml <