# CI/CD Pipeline Reference: Quark's Holo-Grid Ledger

This document defines the complete CI/CD strategy for building, testing, and deploying the Quark's Holo-Grid Ledger monorepo using **Forgejo Actions**, **Hetzner Cloud**, **Docker Compose**, and **OpenTofu**.

---

## 1. Design Principles

| Principle | Rationale |
|---|---|
| **Git-ops driven** | Every infrastructure and deployment change is triggered by a commit, never by manual SSH |
| **Immutable artifacts** | Every merge to `main` produces versioned Docker images stored in the Forgejo Container Registry |
| **Infrastructure-as-Code** | All Hetzner resources (servers, firewalls, networks, volumes) are declared in OpenTofu and applied via pipeline |
| **Separation of concerns** | Infrastructure changes (`infra/`) and application changes (`tongo-core/`, `web-terminal/`) use distinct pipeline stages |
| **Zero-downtime deploy** | Rolling container restarts via `docker compose up -d` with health checks ensure uninterrupted service |

---

## 2. Architecture Overview

The platform uses three isolated OpenTofu workspaces:
1. **`tooling`**: Houses the Forgejo Git host, registry, and runner (Server 1).
2. **`staging`**: Houses the staging application deployment (Server 2 + Volume).
3. **`production`**: Houses the production application deployment (Server 2 + Volume).

This keeps CI/CD workloads isolated from both environments, and ensures staging and production networks are fully partitioned.

```mermaid
graph TB
    subgraph Developer
        DEV[Developer Workstation]
    end

    subgraph "Hetzner Cloud"
        subgraph "tooling Workspace — Server 1 (cx22 · ~€5.49/mo)"
            FORGEJO[Forgejo Instance]
            REG[(Container Registry)]
            RUNNER[Actions Runner + DinD]
        end

        subgraph "staging / production Workspace — Server 2 (cx22 · ~€5.49/mo)"
            CADDY[Caddy Reverse Proxy]
            WEB[Web Terminal Container]
            TONGO[Tongo Core Container]
            VALKEY[(Valkey Cache)]
            VOL[(Hetzner Volume · 20 GB)]
            PRIVNET{{Private Network 10.0.0.0/16}}
        end
    end

    DEV -- git push --> FORGEJO
    FORGEJO -- triggers --> RUNNER
    RUNNER -- docker build + push --> REG
    RUNNER -- SSH deploy --> CADDY
    REG -- docker pull --> WEB
    REG -- docker pull --> TONGO
    CADDY --> WEB
    WEB --> VALKEY
    TONGO --> VALKEY
    VALKEY --> VOL
    CADDY -.- PRIVNET
    VALKEY -.- PRIVNET
```

### Cost Summary

| Resource | Spec | Workspace | Monthly Cost |
|---|---|---|---|
| Forgejo server | CX22 · 2 vCPU · 4 GB RAM | `tooling` | ~€5.49 |
| Application server | CX22 · 2 vCPU · 4 GB RAM | `staging` / `production` | ~€5.49 each |
| Persistent volume | 20 GB ext4 | `staging` / `production` | ~€0.96 each |
| **Total (Staging + Prod + Tools)** | | | **~€18.39** |

---

## 3. Directory Layout

New directories and files added to the monorepo root:

```
quarkshologridledger/
├── .forgejo/
│   └── workflows/
│       ├── ci.yml                     # Lint, test, build on every push/PR
│       ├── deploy.yml                 # Build images, push to registry, deploy on main merge
│       └── infra.yml                  # OpenTofu plan/apply for infrastructure changes
├── infra/                             # OpenTofu infrastructure-as-code
│   ├── versions.tf                    # Provider version constraints (hcloud ~> 1.49)
│   ├── variables.tf                   # Input variables (token, SSH key, domains)
│   ├── network.tf                     # Private network, subnets, firewalls, SSH keys
│   ├── forgejo.tf                     # Forgejo server resource + cloud-init
│   ├── app.tf                         # Application server + volume resources
│   ├── outputs.tf                     # Server IPs, volume ID, network ID
│   ├── cloud-init/
│   │   ├── forgejo.yml                # Bootstrap: Docker + Forgejo + runner compose
│   │   └── app.yml                    # Bootstrap: Docker + volume mount
│   ├── terraform.tfvars.example       # Example variable values (never commit real)
│   └── .gitignore                     # Ignore state files, .terraform/, tfvars
├── deploy/
│   ├── docker-compose.prod.yml        # Production compose (Caddy, Tongo, Web, Valkey)
│   ├── Caddyfile                      # Reverse proxy + automatic HTTPS
│   └── .env.example                   # Template for production env vars
├── tongo-core/
│   └── Dockerfile                     # Multi-stage Go build
├── web-terminal/
│   └── Dockerfile                     # Multi-stage Node build (Vinxi/Nitro)
└── docker-compose.yml                 # Local dev compose (existing — unchanged)
```

---

## 4. Pipeline Stages

### 4.1 CI Pipeline — Every Push & Pull Request

**File:** `.forgejo/workflows/ci.yml`
**Trigger:** All pushes and pull requests to any branch.
**Purpose:** Fast feedback loop — lint, type-check, unit test, and build verification.

```yaml
name: CI

on:
  push:
    branches: ['*']
  pull_request:
    branches: [main]

jobs:
  # ── Go Backend ──────────────────────────────────────────
  tongo-core:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4

      - name: Setup Go
        uses: actions/setup-go@v5
        with:
          go-version: '1.23'

      - name: Lint
        working-directory: tongo-core
        run: |
          go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
          golangci-lint run ./...

      - name: Test
        working-directory: tongo-core
        run: go test -race -coverprofile=coverage.out ./...

      - name: Build
        working-directory: tongo-core
        run: go build -o bin/tongo-core ./cmd/tongo-core

  # ── Frontend ────────────────────────────────────────────
  web-terminal:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: '22'

      - name: Install
        working-directory: web-terminal
        run: npm ci

      - name: Lint & Format Check
        working-directory: web-terminal
        run: npm run format

      - name: Type Check
        working-directory: web-terminal
        run: npx tsc --noEmit

      - name: Build
        working-directory: web-terminal
        run: npm run build
```

> [!NOTE]
> Forgejo Actions uses the same YAML syntax as GitHub Actions. The runner executes
> jobs inside containers via Docker-in-Docker (DinD), so all standard actions are
> compatible.

---

### 4.2 Deploy Pipeline — Merge to Main

**File:** `.forgejo/workflows/deploy.yml`
**Trigger:** Push to `main` branch only.
**Purpose:** Build production Docker images, push to the Forgejo Container Registry, then SSH into the production server and pull + restart.

```yaml
name: Deploy

on:
  push:
    branches: [main]

env:
  REGISTRY: forgejo.yourdomain.com
  IMAGE_TONGO: forgejo.yourdomain.com/costap/quarkshologridledger/tongo-core
  IMAGE_WEB: forgejo.yourdomain.com/costap/quarkshologridledger/web-terminal

jobs:
  build-and-push:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4

      - name: Set image tag
        run: echo "TAG=sha-$(git rev-parse --short HEAD)" >> $GITHUB_ENV

      - name: Login to Forgejo Registry
        run: echo "${{ secrets.REGISTRY_TOKEN }}" | docker login $REGISTRY -u ${{ secrets.REGISTRY_USER }} --password-stdin

      - name: Build & Push Tongo Core
        run: |
          docker build -t $IMAGE_TONGO:$TAG -t $IMAGE_TONGO:latest -f tongo-core/Dockerfile .
          docker push $IMAGE_TONGO:$TAG
          docker push $IMAGE_TONGO:latest

      - name: Build & Push Web Terminal
        run: |
          docker build -t $IMAGE_WEB:$TAG -t $IMAGE_WEB:latest -f web-terminal/Dockerfile .
          docker push $IMAGE_WEB:$TAG
          docker push $IMAGE_WEB:latest

  deploy:
    runs-on: self-hosted
    needs: build-and-push
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to production
        env:
          DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
          DEPLOY_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
        run: |
          mkdir -p ~/.ssh
          echo "$DEPLOY_KEY" > ~/.ssh/deploy_key
          chmod 600 ~/.ssh/deploy_key

          # Copy production compose file
          scp -i ~/.ssh/deploy_key -o StrictHostKeyChecking=no \
            deploy/docker-compose.prod.yml \
            deploy/Caddyfile \
            deploy@$DEPLOY_HOST:/opt/hologrid/

          # Pull new images and restart
          ssh -i ~/.ssh/deploy_key -o StrictHostKeyChecking=no deploy@$DEPLOY_HOST << 'EOF'
            cd /opt/hologrid
            echo "${{ secrets.REGISTRY_TOKEN }}" | docker login ${{ env.REGISTRY }} -u ${{ secrets.REGISTRY_USER }} --password-stdin
            docker compose -f docker-compose.prod.yml pull
            docker compose -f docker-compose.prod.yml up -d --remove-orphans
            docker image prune -f
          EOF
```

> [!IMPORTANT]
> **Secrets required in Forgejo repository settings:**
> - `REGISTRY_TOKEN` — Personal Access Token with `packages:write` scope
> - `REGISTRY_USER` — Forgejo username
> - `DEPLOY_HOST` — Production server IP (from OpenTofu output)
> - `DEPLOY_SSH_KEY` — Private SSH key for deployment user

---

### 4.3 Infrastructure Pipeline — Changes to `infra/`

**File:** `.forgejo/workflows/infra.yml`
**Trigger:** Push to `main` with changes in `infra/**`.
**Purpose:** Plan and apply OpenTofu changes to Hetzner Cloud resources.

```yaml
name: Infrastructure

on:
  push:
    branches: [main]
    paths: ['infra/**']
  pull_request:
    branches: [main]
    paths: ['infra/**']

jobs:
  plan:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4

      - name: Setup OpenTofu
        uses: opentofu/setup-opentofu@v1

      - name: Init
        working-directory: infra
        run: tofu init

      - name: Plan
        working-directory: infra
        env:
          HCLOUD_TOKEN: ${{ secrets.HCLOUD_TOKEN }}
        run: tofu plan -out=tfplan

      - name: Upload plan artifact
        if: github.event_name == 'pull_request'
        uses: actions/upload-artifact@v4
        with:
          name: tfplan
          path: infra/tfplan

  apply:
    runs-on: self-hosted
    needs: plan
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    steps:
      - uses: actions/checkout@v4

      - name: Setup OpenTofu
        uses: opentofu/setup-opentofu@v1

      - name: Init & Apply
        working-directory: infra
        env:
          HCLOUD_TOKEN: ${{ secrets.HCLOUD_TOKEN }}
        run: |
          tofu init
          tofu apply -auto-approve
```

> [!WARNING]
> The `apply` job runs **only** on direct pushes to `main`, never on PRs. PRs
> will show the `plan` output for review. Consider adding a manual approval gate
> for production infrastructure changes as the project matures.

---

## 5. Infrastructure Layer (OpenTofu)

All infrastructure is declared in `infra/` and managed via the `infra.yml` pipeline.
See the scaffolded files for the full implementation.

### 5.1 Resource Map

| Resource | Type | Purpose |
|---|---|---|
| `hcloud_ssh_key.deploy` | SSH Key | Deployment access to both servers |
| `hcloud_network.internal` | Network | Private network `10.0.0.0/16` for inter-server communication |
| `hcloud_network_subnet.servers` | Subnet | `10.0.1.0/24` in `eu-central` zone |
| `hcloud_firewall.forgejo` | Firewall | Forgejo server: allow ports 22, 80, 443 inbound |
| `hcloud_firewall.app` | Firewall | App server: allow ports 22, 80, 443 inbound |
| `hcloud_server.forgejo` | Server (cx22) | Forgejo instance + Actions runner + DinD |
| `hcloud_server.app` | Server (cx22) | Production app stack (Caddy, Tongo, Web, Valkey) |
| `hcloud_volume.data` | Volume (20 GB) | Persistent Valkey data, mounted at `/mnt/data` |
| `hcloud_volume_attachment.data` | Attachment | Binds volume to app server |
| `hcloud_server_network.forgejo` | Network Attach | Forgejo server on private network |
| `hcloud_server_network.app` | Network Attach | App server on private network |

### 5.2 Server Specifications

| Parameter | Forgejo Server | Application Server |
|---|---|---|
| Role | Git forge, CI runner, container registry | Production app stack |
| Type | `cx22` (2 vCPU, 4 GB RAM) | `cx22` (2 vCPU, 4 GB RAM) |
| Image | `ubuntu-24.04` | `ubuntu-24.04` |
| Location | `nbg1` (Nuremberg) | `nbg1` (Nuremberg) |
| Volume | — | 20 GB ext4 at `/mnt/data` |
| Cloud-init | `infra/cloud-init/forgejo.yml` | `infra/cloud-init/app.yml` |

### 5.3 Cloud-Init Bootstrap

Each server is provisioned immutably via `cloud-init` templates in `infra/cloud-init/`.

**Forgejo server (`forgejo.yml`)** — on first boot:
1. Installs Docker Engine + Compose plugin
2. Creates `deploy` user with SSH key + docker group
3. Disables root SSH login
4. Writes a Docker Compose file to `/opt/forgejo/` containing:
   - Forgejo v9 (ports 3000 + 2222, Actions enabled)
   - Docker-in-Docker daemon for the runner
   - Forgejo Actions runner (requires one-time manual registration)
5. Starts Forgejo via `docker compose up -d`

**Application server (`app.yml`)** — on first boot:
1. Installs Docker Engine + Compose plugin
2. Creates `deploy` user with SSH key + docker group
3. Disables root SSH login
4. Formats and mounts the Hetzner Volume at `/mnt/data`
5. Creates `/opt/hologrid/` ready for CI-deployed compose files

> [!NOTE]
> Cloud-init templates use OpenTofu `templatefile()` interpolation.
> Variables like `${ssh_public_key}`, `${forgejo_domain}`, and `${volume_id}`
> are injected at `tofu apply` time.

---

## 6. Production Docker Compose

### 6.1 `deploy/docker-compose.prod.yml`

```yaml
services:
  caddy:
    image: caddy:2-alpine
    container_name: hologrid-caddy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    depends_on:
      web-terminal:
        condition: service_healthy
    restart: unless-stopped

  tongo-core:
    image: forgejo.yourdomain.com/costap/quarkshologridledger/tongo-core:latest
    container_name: hologrid-tongo
    environment:
      - TONGO_CACHE_ADDR=valkey:6379
      - TONGO_LOG_LEVEL=info
    depends_on:
      valkey:
        condition: service_healthy
    restart: unless-stopped

  web-terminal:
    image: forgejo.yourdomain.com/costap/quarkshologridledger/web-terminal:latest
    container_name: hologrid-web
    environment:
      - VALKEY_URL=redis://valkey:6379
      - PORT=3000
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:3000"]
      interval: 10s
      timeout: 5s
      retries: 3
    depends_on:
      valkey:
        condition: service_healthy
    restart: unless-stopped

  valkey:
    image: valkey/valkey:8.0
    container_name: hologrid-valkey
    volumes:
      - /mnt/data/valkey:/data
    healthcheck:
      test: ["CMD", "valkey-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5
    restart: unless-stopped

volumes:
  caddy_data:
  caddy_config:
```

### 6.2 `deploy/Caddyfile`

```caddyfile
hologrid.yourdomain.com {
    reverse_proxy web-terminal:3000
}
```

> [!TIP]
> Caddy automatically provisions and renews Let's Encrypt TLS certificates.
> No certbot configuration required. Just point your DNS A record to the
> Hetzner server IP.

---

## 7. Dockerfiles

### 7.1 `tongo-core/Dockerfile`

```dockerfile
# ── Build Stage ───────────────────────────────────────────
FROM golang:1.23-alpine AS builder

WORKDIR /build
COPY tongo-core/go.mod tongo-core/go.sum ./
RUN go mod download

COPY tongo-core/ .
COPY data-source/ /data-source/

RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /tongo-core ./cmd/tongo-core

# ── Runtime Stage ─────────────────────────────────────────
FROM alpine:3.20

RUN apk --no-cache add ca-certificates
COPY --from=builder /tongo-core /usr/local/bin/tongo-core
COPY --from=builder /data-source /data-source

ENTRYPOINT ["tongo-core"]
CMD ["serve"]
```

### 7.2 `web-terminal/Dockerfile`

```dockerfile
# ── Build Stage ───────────────────────────────────────────
FROM node:22-alpine AS builder

WORKDIR /app
COPY web-terminal/package.json web-terminal/package-lock.json ./
RUN npm ci

COPY web-terminal/ .
RUN npm run build

# ── Runtime Stage ─────────────────────────────────────────
FROM node:22-alpine

WORKDIR /app
COPY --from=builder /app/.output .output
COPY --from=builder /app/package.json .

EXPOSE 3000

CMD ["node", ".output/server/index.mjs"]
```

---

## 8. Secrets & Environment Variables

All secrets are stored in **Forgejo Repository Settings → Actions → Secrets**.

| Secret Name | Description | Used By |
|---|---|---|
| `HCLOUD_TOKEN` | Hetzner Cloud API token | `infra.yml` |
| `REGISTRY_TOKEN` | Forgejo PAT with `packages:write` | `deploy.yml` |
| `REGISTRY_USER` | Forgejo username | `deploy.yml` |
| `DEPLOY_HOST` | Production server IP address | `deploy.yml` |
| `DEPLOY_SSH_KEY` | Private SSH key for `deploy` user | `deploy.yml` |

> [!CAUTION]
> Never commit real tokens, keys, or passwords to the repository. The
> `terraform.tfvars` file is `.gitignore`-d. All sensitive values flow through
> Forgejo Secrets → pipeline environment variables → runtime containers.

---

## 9. Forgejo Runner Setup

The Forgejo Actions runner runs on the **Forgejo server** (Server 1), deployed automatically via cloud-init alongside the Forgejo instance.
The runner uses Docker-in-Docker (DinD) to execute workflow jobs in isolated containers.

### 9.1 Compose Stack (auto-deployed via cloud-init)

The `infra/cloud-init/forgejo.yml` template writes a Docker Compose file to `/opt/forgejo/docker-compose.yml`
containing three services:

| Service | Image | Purpose |
|---|---|---|
| `forgejo` | `codeberg.org/forgejo/forgejo:9` | Git forge, container registry, Actions coordinator |
| `runner-dind` | `docker:dind` | Privileged Docker daemon for runner job execution |
| `runner` | `data.forgejo.org/forgejo/runner:6.2.0` | Actions runner daemon (connects to DinD) |

### 9.2 One-Time Runner Registration

After the Forgejo server boots and the services are running, SSH in and register the runner:

```bash
# SSH into the Forgejo server
ssh deploy@<FORGEJO_IP>

# Enter the runner container
cd /opt/forgejo
docker compose exec runner forgejo-runner register \
  --instance https://forgejo.yourdomain.com \
  --token <REGISTRATION_TOKEN> \
  --name hologrid-runner \
  --labels self-hosted \
  --no-interactive

# Restart to apply config
docker compose restart runner
```

> [!TIP]
> Get the registration token from Forgejo: **Site Administration → Actions → Runners → Create New Runner**.
> This is a one-time operation. The runner persists its config in a Docker volume.

---

## 10. Deployment Flow Summary

```mermaid
sequenceDiagram
    participant Dev as Developer
    participant Forgejo as Forgejo (Server 1)
    participant Runner as CI Runner (Server 1)
    participant Registry as Container Registry (Server 1)
    participant App as App Server (Server 2)

    Dev->>Forgejo: git push (feature branch)
    Forgejo->>Runner: Trigger ci.yml
    Runner->>Runner: Lint + Test + Build (Go & Node)
    Runner-->>Forgejo: ✓ Status check passes

    Dev->>Forgejo: Merge PR → main
    Forgejo->>Runner: Trigger deploy.yml
    Runner->>Runner: docker build (tongo-core, web-terminal)
    Runner->>Registry: docker push :sha-abc1234 + :latest
    Runner->>App: SSH → copy compose + pull + up -d
    App->>Registry: docker pull latest images (via private network)
    App->>App: Rolling restart containers
    App-->>Runner: ✓ Deployment complete
```

---

## 11. Future Enhancements

| Enhancement | Priority | Description |
|---|---|---|
| **Staging environment** | Medium | Add a second Hetzner server for pre-production validation before promoting to prod |
| **Database migrations** | Medium | Add a `migrate` job in `deploy.yml` if persistent SQL storage is added later |
| **Smoke tests** | Medium | Add a post-deploy health check job that curls the production URL and validates HTTP 200 |
| **Notifications** | Low | Integrate Forgejo webhooks to send deployment status to a Matrix/Discord channel |
| **Horizontal scaling** | Low | Move from single server to Docker Swarm or Nomad when traffic demands it |
| **Secrets manager** | Low | Replace Forgejo Secrets with HashiCorp Vault or Infisical for rotation policies |
| **Monitoring stack** | Low | Add Prometheus + Grafana containers to the production compose for observability |
| **Automatic rollback** | Low | If the post-deploy health check fails, automatically roll back to the previous image tag |

---

## 12. Decision Log

| Decision | Chosen | Alternatives Considered | Rationale |
|---|---|---|---|
| **Forgejo hosting** | Self-hosted on Hetzner CX22 (~€5.49/mo) | Codeberg (free, FOSS only), Codey managed (~€39/mo), same-server co-hosting | Cheapest option with full control; separate server isolates CI builds from production; built-in container registry eliminates Docker Hub dependency |
| **Server topology** | Two CX22 servers + private network | Single server for everything, dedicated auction server | Isolation between CI and production workloads; either can be rebuilt independently; total cost ~€12/mo |
| CI/CD engine | Forgejo Actions | Woodpecker CI, Drone | Native integration with Forgejo, GitHub Actions syntax familiarity, built-in container registry |
| Container registry | Forgejo Packages | Harbor, Docker Hub | Zero additional infrastructure — ships with every Forgejo instance |
| IaC tool | OpenTofu | Pulumi, Ansible | Declarative HCL, excellent Hetzner provider, open-source fork of Terraform |
| Reverse proxy | Caddy | Nginx, Traefik | Automatic HTTPS with zero config, single binary, Caddyfile simplicity |
| Server provisioning | cloud-init | Ansible, Packer | No agent required, runs once at boot, perfect for immutable infrastructure |
| Deployment strategy | SSH + docker compose pull | Watchtower, Kubernetes | Simplest approach for a two-server deployment, easy to reason about |
| Persistent storage | Hetzner Volume | Local disk, NFS | Survives server rebuilds, attached to app server only |