holo-grid/quarkshologridledger
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
quarkshologridledger/reference/CICD.md
Pedro Faria da Costa b6bc68de23
Some checks are pending
CI / tongo-core (push) Waiting to run
Details
CI / web-terminal (push) Waiting to run
Details
Deploy / build-and-push (push) Waiting to run
Details
Deploy / deploy (push) Blocked by required conditions
Details
Infrastructure / plan (push) Waiting to run
Details
Infrastructure / apply (push) Blocked by required conditions
Details
add infra
2026-07-05 23:44:51 +01:00

652 lines
22 KiB
Markdown
Raw Blame History

# CI/CD Pipeline Reference: Quark's Holo-Grid Ledger
This document defines the complete CI/CD strategy for building, testing, and deploying the Quark's Holo-Grid Ledger monorepo using **Forgejo Actions**, **Hetzner Cloud**, **Docker Compose**, and **OpenTofu**.
---
## 1. Design Principles
| Principle | Rationale |
|---|---|
| **Git-ops driven** | Every infrastructure and deployment change is triggered by a commit, never by manual SSH |
| **Immutable artifacts** | Every merge to `main` produces versioned Docker images stored in the Forgejo Container Registry |
| **Infrastructure-as-Code** | All Hetzner resources (servers, firewalls, networks, volumes) are declared in OpenTofu and applied via pipeline |
| **Separation of concerns** | Infrastructure changes (`infra/`) and application changes (`tongo-core/`, `web-terminal/`) use distinct pipeline stages |
| **Zero-downtime deploy** | Rolling container restarts via `docker compose up -d` with health checks ensure uninterrupted service |
---
## 2. Architecture Overview
The platform uses three isolated OpenTofu workspaces:
1. **`tooling`**: Houses the Forgejo Git host, registry, and runner (Server 1).
2. **`staging`**: Houses the staging application deployment (Server 2 + Volume).
3. **`production`**: Houses the production application deployment (Server 2 + Volume).
This keeps CI/CD workloads isolated from both environments, and ensures staging and production networks are fully partitioned.
```mermaid
graph TB
subgraph Developer
DEV[Developer Workstation]
end
subgraph "Hetzner Cloud"
subgraph "tooling Workspace — Server 1 (cx22 · ~€5.49/mo)"
FORGEJO[Forgejo Instance]
REG[(Container Registry)]
RUNNER[Actions Runner + DinD]
end
subgraph "staging / production Workspace — Server 2 (cx22 · ~€5.49/mo)"
CADDY[Caddy Reverse Proxy]
WEB[Web Terminal Container]
TONGO[Tongo Core Container]
VALKEY[(Valkey Cache)]
VOL[(Hetzner Volume · 20 GB)]
PRIVNET{{Private Network 10.0.0.0/16}}
end
end
DEV -- git push --> FORGEJO
FORGEJO -- triggers --> RUNNER
RUNNER -- docker build + push --> REG
RUNNER -- SSH deploy --> CADDY
REG -- docker pull --> WEB
REG -- docker pull --> TONGO
CADDY --> WEB
WEB --> VALKEY
TONGO --> VALKEY
VALKEY --> VOL
CADDY -.- PRIVNET
VALKEY -.- PRIVNET
```
### Cost Summary
| Resource | Spec | Workspace | Monthly Cost |
|---|---|---|---|
| Forgejo server | CX22 · 2 vCPU · 4 GB RAM | `tooling` | ~€5.49 |
| Application server | CX22 · 2 vCPU · 4 GB RAM | `staging` / `production` | ~€5.49 each |
| Persistent volume | 20 GB ext4 | `staging` / `production` | ~€0.96 each |
| **Total (Staging + Prod + Tools)** | | | **~€18.39** |
---
## 3. Directory Layout
New directories and files added to the monorepo root:
```
quarkshologridledger/
├── .forgejo/
│ └── workflows/
│ ├── ci.yml # Lint, test, build on every push/PR
│ ├── deploy.yml # Build images, push to registry, deploy on main merge
│ └── infra.yml # OpenTofu plan/apply for infrastructure changes
├── infra/ # OpenTofu infrastructure-as-code
│ ├── versions.tf # Provider version constraints (hcloud ~> 1.49)
│ ├── variables.tf # Input variables (token, SSH key, domains)
│ ├── network.tf # Private network, subnets, firewalls, SSH keys
│ ├── forgejo.tf # Forgejo server resource + cloud-init
│ ├── app.tf # Application server + volume resources
│ ├── outputs.tf # Server IPs, volume ID, network ID
│ ├── cloud-init/
│ │ ├── forgejo.yml # Bootstrap: Docker + Forgejo + runner compose
│ │ └── app.yml # Bootstrap: Docker + volume mount
│ ├── terraform.tfvars.example # Example variable values (never commit real)
│ └── .gitignore # Ignore state files, .terraform/, tfvars
├── deploy/
│ ├── docker-compose.prod.yml # Production compose (Caddy, Tongo, Web, Valkey)
│ ├── Caddyfile # Reverse proxy + automatic HTTPS
│ └── .env.example # Template for production env vars
├── tongo-core/
│ └── Dockerfile # Multi-stage Go build
├── web-terminal/
│ └── Dockerfile # Multi-stage Node build (Vinxi/Nitro)
└── docker-compose.yml # Local dev compose (existing — unchanged)
```
---
## 4. Pipeline Stages
### 4.1 CI Pipeline — Every Push & Pull Request
**File:** `.forgejo/workflows/ci.yml`
**Trigger:** All pushes and pull requests to any branch.
**Purpose:** Fast feedback loop — lint, type-check, unit test, and build verification.
```yaml
name: CI
on:
push:
branches: ['*']
pull_request:
branches: [main]
jobs:
# ── Go Backend ──────────────────────────────────────────
tongo-core:
runs-on: self-hosted
steps:
- uses: actions/checkout@v4
- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: '1.23'
- name: Lint
working-directory: tongo-core
run: |
go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
golangci-lint run ./...
- name: Test
working-directory: tongo-core
run: go test -race -coverprofile=coverage.out ./...
- name: Build
working-directory: tongo-core
run: go build -o bin/tongo-core ./cmd/tongo-core
# ── Frontend ────────────────────────────────────────────
web-terminal:
runs-on: self-hosted
steps:
- uses: actions/checkout@v4
- name: Setup Node
uses: actions/setup-node@v4
with:
node-version: '22'
- name: Install
working-directory: web-terminal
run: npm ci
- name: Lint & Format Check
working-directory: web-terminal
run: npm run format
- name: Type Check
working-directory: web-terminal
run: npx tsc --noEmit
- name: Build
working-directory: web-terminal
run: npm run build
```
> [!NOTE]
> Forgejo Actions uses the same YAML syntax as GitHub Actions. The runner executes
> jobs inside containers via Docker-in-Docker (DinD), so all standard actions are
> compatible.
---
### 4.2 Deploy Pipeline — Merge to Main
**File:** `.forgejo/workflows/deploy.yml`
**Trigger:** Push to `main` branch only.
**Purpose:** Build production Docker images, push to the Forgejo Container Registry, then SSH into the production server and pull + restart.
```yaml
name: Deploy
on:
push:
branches: [main]
env:
REGISTRY: forgejo.yourdomain.com
IMAGE_TONGO: forgejo.yourdomain.com/costap/quarkshologridledger/tongo-core
IMAGE_WEB: forgejo.yourdomain.com/costap/quarkshologridledger/web-terminal
jobs:
build-and-push:
runs-on: self-hosted
steps:
- uses: actions/checkout@v4
- name: Set image tag
run: echo "TAG=sha-$(git rev-parse --short HEAD)" >> $GITHUB_ENV
- name: Login to Forgejo Registry
run: echo "${{ secrets.REGISTRY_TOKEN }}" | docker login $REGISTRY -u ${{ secrets.REGISTRY_USER }} --password-stdin
- name: Build & Push Tongo Core
run: |
docker build -t $IMAGE_TONGO:$TAG -t $IMAGE_TONGO:latest -f tongo-core/Dockerfile .
docker push $IMAGE_TONGO:$TAG
docker push $IMAGE_TONGO:latest
- name: Build & Push Web Terminal
run: |
docker build -t $IMAGE_WEB:$TAG -t $IMAGE_WEB:latest -f web-terminal/Dockerfile .
docker push $IMAGE_WEB:$TAG
docker push $IMAGE_WEB:latest
deploy:
runs-on: self-hosted
needs: build-and-push
steps:
- uses: actions/checkout@v4
- name: Deploy to production
env:
DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
DEPLOY_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
run: |
mkdir -p ~/.ssh
echo "$DEPLOY_KEY" > ~/.ssh/deploy_key
chmod 600 ~/.ssh/deploy_key
# Copy production compose file
scp -i ~/.ssh/deploy_key -o StrictHostKeyChecking=no \
deploy/docker-compose.prod.yml \
deploy/Caddyfile \
deploy@$DEPLOY_HOST:/opt/hologrid/
# Pull new images and restart
ssh -i ~/.ssh/deploy_key -o StrictHostKeyChecking=no deploy@$DEPLOY_HOST << 'EOF'
cd /opt/hologrid
echo "${{ secrets.REGISTRY_TOKEN }}" | docker login ${{ env.REGISTRY }} -u ${{ secrets.REGISTRY_USER }} --password-stdin
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d --remove-orphans
docker image prune -f
EOF
```
> [!IMPORTANT]
> **Secrets required in Forgejo repository settings:**
> - `REGISTRY_TOKEN` — Personal Access Token with `packages:write` scope
> - `REGISTRY_USER` — Forgejo username
> - `DEPLOY_HOST` — Production server IP (from OpenTofu output)
> - `DEPLOY_SSH_KEY` — Private SSH key for deployment user
---
### 4.3 Infrastructure Pipeline — Changes to `infra/`
**File:** `.forgejo/workflows/infra.yml`
**Trigger:** Push to `main` with changes in `infra/**`.
**Purpose:** Plan and apply OpenTofu changes to Hetzner Cloud resources.
```yaml
name: Infrastructure
on:
push:
branches: [main]
paths: ['infra/**']
pull_request:
branches: [main]
paths: ['infra/**']
jobs:
plan:
runs-on: self-hosted
steps:
- uses: actions/checkout@v4
- name: Setup OpenTofu
uses: opentofu/setup-opentofu@v1
- name: Init
working-directory: infra
run: tofu init
- name: Plan
working-directory: infra
env:
HCLOUD_TOKEN: ${{ secrets.HCLOUD_TOKEN }}
run: tofu plan -out=tfplan
- name: Upload plan artifact
if: github.event_name == 'pull_request'
uses: actions/upload-artifact@v4
with:
name: tfplan
path: infra/tfplan
apply:
runs-on: self-hosted
needs: plan
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
steps:
- uses: actions/checkout@v4
- name: Setup OpenTofu
uses: opentofu/setup-opentofu@v1
- name: Init & Apply
working-directory: infra
env:
HCLOUD_TOKEN: ${{ secrets.HCLOUD_TOKEN }}
run: |
tofu init
tofu apply -auto-approve
```
> [!WARNING]
> The `apply` job runs **only** on direct pushes to `main`, never on PRs. PRs
> will show the `plan` output for review. Consider adding a manual approval gate
> for production infrastructure changes as the project matures.
---
## 5. Infrastructure Layer (OpenTofu)
All infrastructure is declared in `infra/` and managed via the `infra.yml` pipeline.
See the scaffolded files for the full implementation.
### 5.1 Resource Map
| Resource | Type | Purpose |
|---|---|---|
| `hcloud_ssh_key.deploy` | SSH Key | Deployment access to both servers |
| `hcloud_network.internal` | Network | Private network `10.0.0.0/16` for inter-server communication |
| `hcloud_network_subnet.servers` | Subnet | `10.0.1.0/24` in `eu-central` zone |
| `hcloud_firewall.forgejo` | Firewall | Forgejo server: allow ports 22, 80, 443 inbound |
| `hcloud_firewall.app` | Firewall | App server: allow ports 22, 80, 443 inbound |
| `hcloud_server.forgejo` | Server (cx22) | Forgejo instance + Actions runner + DinD |
| `hcloud_server.app` | Server (cx22) | Production app stack (Caddy, Tongo, Web, Valkey) |
| `hcloud_volume.data` | Volume (20 GB) | Persistent Valkey data, mounted at `/mnt/data` |
| `hcloud_volume_attachment.data` | Attachment | Binds volume to app server |
| `hcloud_server_network.forgejo` | Network Attach | Forgejo server on private network |
| `hcloud_server_network.app` | Network Attach | App server on private network |
### 5.2 Server Specifications
| Parameter | Forgejo Server | Application Server |
|---|---|---|
| Role | Git forge, CI runner, container registry | Production app stack |
| Type | `cx22` (2 vCPU, 4 GB RAM) | `cx22` (2 vCPU, 4 GB RAM) |
| Image | `ubuntu-24.04` | `ubuntu-24.04` |
| Location | `nbg1` (Nuremberg) | `nbg1` (Nuremberg) |
| Volume | — | 20 GB ext4 at `/mnt/data` |
| Cloud-init | `infra/cloud-init/forgejo.yml` | `infra/cloud-init/app.yml` |
### 5.3 Cloud-Init Bootstrap
Each server is provisioned immutably via `cloud-init` templates in `infra/cloud-init/`.
**Forgejo server (`forgejo.yml`)** — on first boot:
1. Installs Docker Engine + Compose plugin
2. Creates `deploy` user with SSH key + docker group
3. Disables root SSH login
4. Writes a Docker Compose file to `/opt/forgejo/` containing:
- Forgejo v9 (ports 3000 + 2222, Actions enabled)
- Docker-in-Docker daemon for the runner
- Forgejo Actions runner (requires one-time manual registration)
5. Starts Forgejo via `docker compose up -d`
**Application server (`app.yml`)** — on first boot:
1. Installs Docker Engine + Compose plugin
2. Creates `deploy` user with SSH key + docker group
3. Disables root SSH login
4. Formats and mounts the Hetzner Volume at `/mnt/data`
5. Creates `/opt/hologrid/` ready for CI-deployed compose files
> [!NOTE]
> Cloud-init templates use OpenTofu `templatefile()` interpolation.
> Variables like `${ssh_public_key}`, `${forgejo_domain}`, and `${volume_id}`
> are injected at `tofu apply` time.
---
## 6. Production Docker Compose
### 6.1 `deploy/docker-compose.prod.yml`
```yaml
services:
caddy:
image: caddy:2-alpine
container_name: hologrid-caddy
ports:
- "80:80"
- "443:443"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- caddy_data:/data
- caddy_config:/config
depends_on:
web-terminal:
condition: service_healthy
restart: unless-stopped
tongo-core:
image: forgejo.yourdomain.com/costap/quarkshologridledger/tongo-core:latest
container_name: hologrid-tongo
environment:
- TONGO_CACHE_ADDR=valkey:6379
- TONGO_LOG_LEVEL=info
depends_on:
valkey:
condition: service_healthy
restart: unless-stopped
web-terminal:
image: forgejo.yourdomain.com/costap/quarkshologridledger/web-terminal:latest
container_name: hologrid-web
environment:
- VALKEY_URL=redis://valkey:6379
- PORT=3000
healthcheck:
test: ["CMD", "wget", "--spider", "-q", "http://localhost:3000"]
interval: 10s
timeout: 5s
retries: 3
depends_on:
valkey:
condition: service_healthy
restart: unless-stopped
valkey:
image: valkey/valkey:8.0
container_name: hologrid-valkey
volumes:
- /mnt/data/valkey:/data
healthcheck:
test: ["CMD", "valkey-cli", "ping"]
interval: 5s
timeout: 3s
retries: 5
restart: unless-stopped
volumes:
caddy_data:
caddy_config:
```
### 6.2 `deploy/Caddyfile`
```caddyfile
hologrid.yourdomain.com {
reverse_proxy web-terminal:3000
}
```
> [!TIP]
> Caddy automatically provisions and renews Let's Encrypt TLS certificates.
> No certbot configuration required. Just point your DNS A record to the
> Hetzner server IP.
---
## 7. Dockerfiles
### 7.1 `tongo-core/Dockerfile`
```dockerfile
# ── Build Stage ───────────────────────────────────────────
FROM golang:1.23-alpine AS builder
WORKDIR /build
COPY tongo-core/go.mod tongo-core/go.sum ./
RUN go mod download
COPY tongo-core/ .
COPY data-source/ /data-source/
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /tongo-core ./cmd/tongo-core
# ── Runtime Stage ─────────────────────────────────────────
FROM alpine:3.20
RUN apk --no-cache add ca-certificates
COPY --from=builder /tongo-core /usr/local/bin/tongo-core
COPY --from=builder /data-source /data-source
ENTRYPOINT ["tongo-core"]
CMD ["serve"]
```
### 7.2 `web-terminal/Dockerfile`
```dockerfile
# ── Build Stage ───────────────────────────────────────────
FROM node:22-alpine AS builder
WORKDIR /app
COPY web-terminal/package.json web-terminal/package-lock.json ./
RUN npm ci
COPY web-terminal/ .
RUN npm run build
# ── Runtime Stage ─────────────────────────────────────────
FROM node:22-alpine
WORKDIR /app
COPY --from=builder /app/.output .output
COPY --from=builder /app/package.json .
EXPOSE 3000
CMD ["node", ".output/server/index.mjs"]
```
---
## 8. Secrets & Environment Variables
All secrets are stored in **Forgejo Repository Settings → Actions → Secrets**.
| Secret Name | Description | Used By |
|---|---|---|
| `HCLOUD_TOKEN` | Hetzner Cloud API token | `infra.yml` |
| `REGISTRY_TOKEN` | Forgejo PAT with `packages:write` | `deploy.yml` |
| `REGISTRY_USER` | Forgejo username | `deploy.yml` |
| `DEPLOY_HOST` | Production server IP address | `deploy.yml` |
| `DEPLOY_SSH_KEY` | Private SSH key for `deploy` user | `deploy.yml` |
> [!CAUTION]
> Never commit real tokens, keys, or passwords to the repository. The
> `terraform.tfvars` file is `.gitignore`-d. All sensitive values flow through
> Forgejo Secrets → pipeline environment variables → runtime containers.
---
## 9. Forgejo Runner Setup
The Forgejo Actions runner runs on the **Forgejo server** (Server 1), deployed automatically via cloud-init alongside the Forgejo instance.
The runner uses Docker-in-Docker (DinD) to execute workflow jobs in isolated containers.
### 9.1 Compose Stack (auto-deployed via cloud-init)
The `infra/cloud-init/forgejo.yml` template writes a Docker Compose file to `/opt/forgejo/docker-compose.yml`
containing three services:
| Service | Image | Purpose |
|---|---|---|
| `forgejo` | `codeberg.org/forgejo/forgejo:9` | Git forge, container registry, Actions coordinator |
| `runner-dind` | `docker:dind` | Privileged Docker daemon for runner job execution |
| `runner` | `data.forgejo.org/forgejo/runner:6.2.0` | Actions runner daemon (connects to DinD) |
### 9.2 One-Time Runner Registration
After the Forgejo server boots and the services are running, SSH in and register the runner:
```bash
# SSH into the Forgejo server
ssh deploy@<FORGEJO_IP>
# Enter the runner container
cd /opt/forgejo
docker compose exec runner forgejo-runner register \
--instance https://forgejo.yourdomain.com \
--token <REGISTRATION_TOKEN> \
--name hologrid-runner \
--labels self-hosted \
--no-interactive
# Restart to apply config
docker compose restart runner
```
> [!TIP]
> Get the registration token from Forgejo: **Site Administration → Actions → Runners → Create New Runner**.
> This is a one-time operation. The runner persists its config in a Docker volume.
---
## 10. Deployment Flow Summary
```mermaid
sequenceDiagram
participant Dev as Developer
participant Forgejo as Forgejo (Server 1)
participant Runner as CI Runner (Server 1)
participant Registry as Container Registry (Server 1)
participant App as App Server (Server 2)
Dev->>Forgejo: git push (feature branch)
Forgejo->>Runner: Trigger ci.yml
Runner->>Runner: Lint + Test + Build (Go & Node)
Runner-->>Forgejo: ✓ Status check passes
Dev->>Forgejo: Merge PR → main
Forgejo->>Runner: Trigger deploy.yml
Runner->>Runner: docker build (tongo-core, web-terminal)
Runner->>Registry: docker push :sha-abc1234 + :latest
Runner->>App: SSH → copy compose + pull + up -d
App->>Registry: docker pull latest images (via private network)
App->>App: Rolling restart containers
App-->>Runner: ✓ Deployment complete
```
---
## 11. Future Enhancements
| Enhancement | Priority | Description |
|---|---|---|
| **Staging environment** | Medium | Add a second Hetzner server for pre-production validation before promoting to prod |
| **Database migrations** | Medium | Add a `migrate` job in `deploy.yml` if persistent SQL storage is added later |
| **Smoke tests** | Medium | Add a post-deploy health check job that curls the production URL and validates HTTP 200 |
| **Notifications** | Low | Integrate Forgejo webhooks to send deployment status to a Matrix/Discord channel |
| **Horizontal scaling** | Low | Move from single server to Docker Swarm or Nomad when traffic demands it |
| **Secrets manager** | Low | Replace Forgejo Secrets with HashiCorp Vault or Infisical for rotation policies |
| **Monitoring stack** | Low | Add Prometheus + Grafana containers to the production compose for observability |
| **Automatic rollback** | Low | If the post-deploy health check fails, automatically roll back to the previous image tag |
---
## 12. Decision Log
| Decision | Chosen | Alternatives Considered | Rationale |
|---|---|---|---|
| **Forgejo hosting** | Self-hosted on Hetzner CX22 (~€5.49/mo) | Codeberg (free, FOSS only), Codey managed (~€39/mo), same-server co-hosting | Cheapest option with full control; separate server isolates CI builds from production; built-in container registry eliminates Docker Hub dependency |
| **Server topology** | Two CX22 servers + private network | Single server for everything, dedicated auction server | Isolation between CI and production workloads; either can be rebuilt independently; total cost ~€12/mo |
| CI/CD engine | Forgejo Actions | Woodpecker CI, Drone | Native integration with Forgejo, GitHub Actions syntax familiarity, built-in container registry |
| Container registry | Forgejo Packages | Harbor, Docker Hub | Zero additional infrastructure — ships with every Forgejo instance |
| IaC tool | OpenTofu | Pulumi, Ansible | Declarative HCL, excellent Hetzner provider, open-source fork of Terraform |
| Reverse proxy | Caddy | Nginx, Traefik | Automatic HTTPS with zero config, single binary, Caddyfile simplicity |
| Server provisioning | cloud-init | Ansible, Packer | No agent required, runs once at boot, perfect for immutable infrastructure |
| Deployment strategy | SSH + docker compose pull | Watchtower, Kubernetes | Simplest approach for a two-server deployment, easy to reason about |
| Persistent storage | Hetzner Volume | Local disk, NFS | Survives server rebuilds, attached to app server only |
Reference in a new issue View git blame Copy permalink