quarkshologridledger/reference/CICD.md

CI/CD Pipeline Reference: Quark's Holo-Grid Ledger

This document defines the complete CI/CD strategy for building, testing, and deploying the Quark's Holo-Grid Ledger monorepo using Forgejo Actions, Hetzner Cloud, Docker Compose, and OpenTofu.

---

1. Design Principles

Principle	Rationale
Git-ops driven	Every infrastructure and deployment change is triggered by a commit, never by manual SSH
Immutable artifacts	Every merge to main produces versioned Docker images stored in the Forgejo Container Registry
Infrastructure-as-Code	All Hetzner resources (servers, firewalls, networks, volumes) are declared in OpenTofu and applied via pipeline
Separation of concerns	Infrastructure changes (infra/) and application changes (tongo-core/, web-terminal/) use distinct pipeline stages
Zero-downtime deploy	Rolling container restarts via docker compose up -d with health checks ensure uninterrupted service

---

2. Architecture Overview

The platform uses three isolated OpenTofu workspaces:

1. tooling: Houses the Forgejo Git host, registry, and runner (Server 1).
2. staging: Houses the staging application deployment (Server 2 + Volume).
3. production: Houses the production application deployment (Server 2 + Volume).

This keeps CI/CD workloads isolated from both environments, and ensures staging and production networks are fully partitioned.

graph TB
    subgraph Developer
        DEV[Developer Workstation]
    end

    subgraph "Hetzner Cloud"
        subgraph "tooling Workspace — Server 1 (cx22 · ~€5.49/mo)"
            FORGEJO[Forgejo Instance]
            REG[(Container Registry)]
            RUNNER[Actions Runner + DinD]
        end

        subgraph "staging / production Workspace — Server 2 (cx22 · ~€5.49/mo)"
            CADDY[Caddy Reverse Proxy]
            WEB[Web Terminal Container]
            TONGO[Tongo Core Container]
            VALKEY[(Valkey Cache)]
            VOL[(Hetzner Volume · 20 GB)]
            PRIVNET{{Private Network 10.0.0.0/16}}
        end
    end

    DEV -- git push --> FORGEJO
    FORGEJO -- triggers --> RUNNER
    RUNNER -- docker build + push --> REG
    RUNNER -- SSH deploy --> CADDY
    REG -- docker pull --> WEB
    REG -- docker pull --> TONGO
    CADDY --> WEB
    WEB --> VALKEY
    TONGO --> VALKEY
    VALKEY --> VOL
    CADDY -.- PRIVNET
    VALKEY -.- PRIVNET

Cost Summary

Resource	Spec	Workspace	Monthly Cost
Forgejo server	CX22 · 2 vCPU · 4 GB RAM	tooling	~€5.49
Application server	CX22 · 2 vCPU · 4 GB RAM	staging / production	~€5.49 each
Persistent volume	20 GB ext4	staging / production	~€0.96 each
Total (Staging + Prod + Tools)			~€18.39

---

3. Directory Layout

New directories and files added to the monorepo root:

quarkshologridledger/
├── .forgejo/
│   └── workflows/
│       ├── ci.yml                     # Lint, test, build on every push/PR
│       ├── deploy.yml                 # Build images, push to registry, deploy on main merge
│       └── infra.yml                  # OpenTofu plan/apply for infrastructure changes
├── infra/                             # OpenTofu infrastructure-as-code
│   ├── versions.tf                    # Provider version constraints (hcloud ~> 1.49)
│   ├── variables.tf                   # Input variables (token, SSH key, domains)
│   ├── network.tf                     # Private network, subnets, firewalls, SSH keys
│   ├── forgejo.tf                     # Forgejo server resource + cloud-init
│   ├── app.tf                         # Application server + volume resources
│   ├── outputs.tf                     # Server IPs, volume ID, network ID
│   ├── cloud-init/
│   │   ├── forgejo.yml                # Bootstrap: Docker + Forgejo + runner compose
│   │   └── app.yml                    # Bootstrap: Docker + volume mount
│   ├── terraform.tfvars.example       # Example variable values (never commit real)
│   └── .gitignore                     # Ignore state files, .terraform/, tfvars
├── deploy/
│   ├── docker-compose.prod.yml        # Production compose (Caddy, Tongo, Web, Valkey)
│   ├── Caddyfile                      # Reverse proxy + automatic HTTPS
│   └── .env.example                   # Template for production env vars
├── tongo-core/
│   └── Dockerfile                     # Multi-stage Go build
├── web-terminal/
│   └── Dockerfile                     # Multi-stage Node build (Vinxi/Nitro)
└── docker-compose.yml                 # Local dev compose (existing — unchanged)

---

4. Pipeline Stages

4.1 CI Pipeline — Every Push & Pull Request

File: .forgejo/workflows/ci.yml Trigger: All pushes and pull requests to any branch. Purpose: Fast feedback loop — lint, type-check, unit test, and build verification.

name: CI

on:
  push:
    branches: ['*']
  pull_request:
    branches: [main]

jobs:
  # ── Go Backend ──────────────────────────────────────────
  tongo-core:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4

      - name: Setup Go
        uses: actions/setup-go@v5
        with:
          go-version: '1.23'

      - name: Lint
        working-directory: tongo-core
        run: |
          go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest
          golangci-lint run ./...          

      - name: Test
        working-directory: tongo-core
        run: go test -race -coverprofile=coverage.out ./...

      - name: Build
        working-directory: tongo-core
        run: go build -o bin/tongo-core ./cmd/tongo-core

  # ── Frontend ────────────────────────────────────────────
  web-terminal:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: '22'

      - name: Install
        working-directory: web-terminal
        run: npm ci

      - name: Lint & Format Check
        working-directory: web-terminal
        run: npm run format

      - name: Type Check
        working-directory: web-terminal
        run: npx tsc --noEmit

      - name: Build
        working-directory: web-terminal
        run: npm run build

Note

Forgejo Actions uses the same YAML syntax as GitHub Actions. The runner executes jobs inside containers via Docker-in-Docker (DinD), so all standard actions are compatible.

---

4.2 Deploy Pipeline — Merge to Main

File: .forgejo/workflows/deploy.yml Trigger: Push to main branch only. Purpose: Build production Docker images, push to the Forgejo Container Registry, then SSH into the production server and pull + restart.

name: Deploy

on:
  push:
    branches: [main]

env:
  REGISTRY: forgejo.yourdomain.com
  IMAGE_TONGO: forgejo.yourdomain.com/costap/quarkshologridledger/tongo-core
  IMAGE_WEB: forgejo.yourdomain.com/costap/quarkshologridledger/web-terminal

jobs:
  build-and-push:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4

      - name: Set image tag
        run: echo "TAG=sha-$(git rev-parse --short HEAD)" >> $GITHUB_ENV

      - name: Login to Forgejo Registry
        run: echo "${{ secrets.REGISTRY_TOKEN }}" | docker login $REGISTRY -u ${{ secrets.REGISTRY_USER }} --password-stdin

      - name: Build & Push Tongo Core
        run: |
          docker build -t $IMAGE_TONGO:$TAG -t $IMAGE_TONGO:latest -f tongo-core/Dockerfile .
          docker push $IMAGE_TONGO:$TAG
          docker push $IMAGE_TONGO:latest          

      - name: Build & Push Web Terminal
        run: |
          docker build -t $IMAGE_WEB:$TAG -t $IMAGE_WEB:latest -f web-terminal/Dockerfile .
          docker push $IMAGE_WEB:$TAG
          docker push $IMAGE_WEB:latest          

  deploy:
    runs-on: self-hosted
    needs: build-and-push
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to production
        env:
          DEPLOY_HOST: ${{ secrets.DEPLOY_HOST }}
          DEPLOY_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
        run: |
          mkdir -p ~/.ssh
          echo "$DEPLOY_KEY" > ~/.ssh/deploy_key
          chmod 600 ~/.ssh/deploy_key

          # Copy production compose file
          scp -i ~/.ssh/deploy_key -o StrictHostKeyChecking=no \
            deploy/docker-compose.prod.yml \
            deploy/Caddyfile \
            deploy@$DEPLOY_HOST:/opt/hologrid/

          # Pull new images and restart
          ssh -i ~/.ssh/deploy_key -o StrictHostKeyChecking=no deploy@$DEPLOY_HOST << 'EOF'
            cd /opt/hologrid
            echo "${{ secrets.REGISTRY_TOKEN }}" | docker login ${{ env.REGISTRY }} -u ${{ secrets.REGISTRY_USER }} --password-stdin
            docker compose -f docker-compose.prod.yml pull
            docker compose -f docker-compose.prod.yml up -d --remove-orphans
            docker image prune -f
          EOF          

Important

Secrets required in Forgejo repository settings:

• REGISTRY_TOKEN — Personal Access Token with packages:write scope
• REGISTRY_USER — Forgejo username
• DEPLOY_HOST — Production server IP (from OpenTofu output)
• DEPLOY_SSH_KEY — Private SSH key for deployment user

---

4.3 Infrastructure Pipeline — Changes to infra/

File: .forgejo/workflows/infra.yml Trigger: Push to main with changes in infra/**. Purpose: Plan and apply OpenTofu changes to Hetzner Cloud resources.

name: Infrastructure

on:
  push:
    branches: [main]
    paths: ['infra/**']
  pull_request:
    branches: [main]
    paths: ['infra/**']

jobs:
  plan:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4

      - name: Setup OpenTofu
        uses: opentofu/setup-opentofu@v1

      - name: Init
        working-directory: infra
        run: tofu init

      - name: Plan
        working-directory: infra
        env:
          HCLOUD_TOKEN: ${{ secrets.HCLOUD_TOKEN }}
        run: tofu plan -out=tfplan

      - name: Upload plan artifact
        if: github.event_name == 'pull_request'
        uses: actions/upload-artifact@v4
        with:
          name: tfplan
          path: infra/tfplan

  apply:
    runs-on: self-hosted
    needs: plan
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    steps:
      - uses: actions/checkout@v4

      - name: Setup OpenTofu
        uses: opentofu/setup-opentofu@v1

      - name: Init & Apply
        working-directory: infra
        env:
          HCLOUD_TOKEN: ${{ secrets.HCLOUD_TOKEN }}
        run: |
          tofu init
          tofu apply -auto-approve          

Warning

The apply job runs only on direct pushes to main, never on PRs. PRs will show the plan output for review. Consider adding a manual approval gate for production infrastructure changes as the project matures.

---

5. Infrastructure Layer (OpenTofu)

All infrastructure is declared in infra/ and managed via the infra.yml pipeline. See the scaffolded files for the full implementation.

5.1 Resource Map

Resource	Type	Purpose
hcloud_ssh_key.deploy	SSH Key	Deployment access to both servers
hcloud_network.internal	Network	Private network 10.0.0.0/16 for inter-server communication
hcloud_network_subnet.servers	Subnet	10.0.1.0/24 in eu-central zone
hcloud_firewall.forgejo	Firewall	Forgejo server: allow ports 22, 80, 443 inbound
hcloud_firewall.app	Firewall	App server: allow ports 22, 80, 443 inbound
hcloud_server.forgejo	Server (cx22)	Forgejo instance + Actions runner + DinD
hcloud_server.app	Server (cx22)	Production app stack (Caddy, Tongo, Web, Valkey)
hcloud_volume.data	Volume (20 GB)	Persistent Valkey data, mounted at /mnt/data
hcloud_volume_attachment.data	Attachment	Binds volume to app server
hcloud_server_network.forgejo	Network Attach	Forgejo server on private network
hcloud_server_network.app	Network Attach	App server on private network

5.2 Server Specifications

Parameter	Forgejo Server	Application Server
Role	Git forge, CI runner, container registry	Production app stack
Type	cx22 (2 vCPU, 4 GB RAM)	cx22 (2 vCPU, 4 GB RAM)
Image	ubuntu-24.04	ubuntu-24.04
Location	nbg1 (Nuremberg)	nbg1 (Nuremberg)
Volume	—	20 GB ext4 at /mnt/data
Cloud-init	infra/cloud-init/forgejo.yml	infra/cloud-init/app.yml

5.3 Cloud-Init Bootstrap

Each server is provisioned immutably via cloud-init templates in infra/cloud-init/.

Forgejo server (forgejo.yml) — on first boot:

1. Installs Docker Engine + Compose plugin
2. Creates deploy user with SSH key + docker group
3. Disables root SSH login
4. Writes a Docker Compose file to /opt/forgejo/ containing:
   • Forgejo v9 (ports 3000 + 2222, Actions enabled)
   • Docker-in-Docker daemon for the runner
   • Forgejo Actions runner (requires one-time manual registration)
5. Starts Forgejo via docker compose up -d

Application server (app.yml) — on first boot:

1. Installs Docker Engine + Compose plugin
2. Creates deploy user with SSH key + docker group
3. Disables root SSH login
4. Formats and mounts the Hetzner Volume at /mnt/data
5. Creates /opt/hologrid/ ready for CI-deployed compose files

Note

Cloud-init templates use OpenTofu templatefile() interpolation. Variables like ${ssh_public_key}, ${forgejo_domain}, and ${volume_id} are injected at tofu apply time.

---

6. Production Docker Compose

6.1 deploy/docker-compose.prod.yml

services:
  caddy:
    image: caddy:2-alpine
    container_name: hologrid-caddy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    depends_on:
      web-terminal:
        condition: service_healthy
    restart: unless-stopped

  tongo-core:
    image: forgejo.yourdomain.com/costap/quarkshologridledger/tongo-core:latest
    container_name: hologrid-tongo
    environment:
      - TONGO_CACHE_ADDR=valkey:6379
      - TONGO_LOG_LEVEL=info
    depends_on:
      valkey:
        condition: service_healthy
    restart: unless-stopped

  web-terminal:
    image: forgejo.yourdomain.com/costap/quarkshologridledger/web-terminal:latest
    container_name: hologrid-web
    environment:
      - VALKEY_URL=redis://valkey:6379
      - PORT=3000
    healthcheck:
      test: ["CMD", "wget", "--spider", "-q", "http://localhost:3000"]
      interval: 10s
      timeout: 5s
      retries: 3
    depends_on:
      valkey:
        condition: service_healthy
    restart: unless-stopped

  valkey:
    image: valkey/valkey:8.0
    container_name: hologrid-valkey
    volumes:
      - /mnt/data/valkey:/data
    healthcheck:
      test: ["CMD", "valkey-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 5
    restart: unless-stopped

volumes:
  caddy_data:
  caddy_config:

6.2 deploy/Caddyfile

hologrid.yourdomain.com {
    reverse_proxy web-terminal:3000
}

Tip

Caddy automatically provisions and renews Let's Encrypt TLS certificates. No certbot configuration required. Just point your DNS A record to the Hetzner server IP.

---

7. Dockerfiles

7.1 tongo-core/Dockerfile

# ── Build Stage ───────────────────────────────────────────
FROM golang:1.23-alpine AS builder

WORKDIR /build
COPY tongo-core/go.mod tongo-core/go.sum ./
RUN go mod download

COPY tongo-core/ .
COPY data-source/ /data-source/

RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /tongo-core ./cmd/tongo-core

# ── Runtime Stage ─────────────────────────────────────────
FROM alpine:3.20

RUN apk --no-cache add ca-certificates
COPY --from=builder /tongo-core /usr/local/bin/tongo-core
COPY --from=builder /data-source /data-source

ENTRYPOINT ["tongo-core"]
CMD ["serve"]

7.2 web-terminal/Dockerfile

# ── Build Stage ───────────────────────────────────────────
FROM node:22-alpine AS builder

WORKDIR /app
COPY web-terminal/package.json web-terminal/package-lock.json ./
RUN npm ci

COPY web-terminal/ .
RUN npm run build

# ── Runtime Stage ─────────────────────────────────────────
FROM node:22-alpine

WORKDIR /app
COPY --from=builder /app/.output .output
COPY --from=builder /app/package.json .

EXPOSE 3000

CMD ["node", ".output/server/index.mjs"]

---

8. Secrets & Environment Variables

All secrets are stored in Forgejo Repository Settings → Actions → Secrets.

Secret Name	Description	Used By
HCLOUD_TOKEN	Hetzner Cloud API token	infra.yml
REGISTRY_TOKEN	Forgejo PAT with packages:write	deploy.yml
REGISTRY_USER	Forgejo username	deploy.yml
DEPLOY_HOST	Production server IP address	deploy.yml
DEPLOY_SSH_KEY	Private SSH key for deploy user	deploy.yml

Caution

Never commit real tokens, keys, or passwords to the repository. The terraform.tfvars file is .gitignore-d. All sensitive values flow through Forgejo Secrets → pipeline environment variables → runtime containers.

---

9. Forgejo Runner Setup

The Forgejo Actions runner runs on the Forgejo server (Server 1), deployed automatically via cloud-init alongside the Forgejo instance. The runner uses Docker-in-Docker (DinD) to execute workflow jobs in isolated containers.

9.1 Compose Stack (auto-deployed via cloud-init)

The infra/cloud-init/forgejo.yml template writes a Docker Compose file to /opt/forgejo/docker-compose.yml containing three services:

Service	Image	Purpose
forgejo	codeberg.org/forgejo/forgejo:9	Git forge, container registry, Actions coordinator
runner-dind	docker:dind	Privileged Docker daemon for runner job execution
runner	data.forgejo.org/forgejo/runner:6.2.0	Actions runner daemon (connects to DinD)

9.2 One-Time Runner Registration

After the Forgejo server boots and the services are running, SSH in and register the runner:

# SSH into the Forgejo server
ssh deploy@<FORGEJO_IP>

# Enter the runner container
cd /opt/forgejo
docker compose exec runner forgejo-runner register \
  --instance https://forgejo.yourdomain.com \
  --token <REGISTRATION_TOKEN> \
  --name hologrid-runner \
  --labels self-hosted \
  --no-interactive

# Restart to apply config
docker compose restart runner

Tip

Get the registration token from Forgejo: Site Administration → Actions → Runners → Create New Runner. This is a one-time operation. The runner persists its config in a Docker volume.

---

10. Deployment Flow Summary

sequenceDiagram
    participant Dev as Developer
    participant Forgejo as Forgejo (Server 1)
    participant Runner as CI Runner (Server 1)
    participant Registry as Container Registry (Server 1)
    participant App as App Server (Server 2)

    Dev->>Forgejo: git push (feature branch)
    Forgejo->>Runner: Trigger ci.yml
    Runner->>Runner: Lint + Test + Build (Go & Node)
    Runner-->>Forgejo: ✓ Status check passes

    Dev->>Forgejo: Merge PR → main
    Forgejo->>Runner: Trigger deploy.yml
    Runner->>Runner: docker build (tongo-core, web-terminal)
    Runner->>Registry: docker push :sha-abc1234 + :latest
    Runner->>App: SSH → copy compose + pull + up -d
    App->>Registry: docker pull latest images (via private network)
    App->>App: Rolling restart containers
    App-->>Runner: ✓ Deployment complete

---

11. Future Enhancements

Enhancement	Priority	Description
Staging environment	Medium	Add a second Hetzner server for pre-production validation before promoting to prod
Database migrations	Medium	Add a migrate job in deploy.yml if persistent SQL storage is added later
Smoke tests	Medium	Add a post-deploy health check job that curls the production URL and validates HTTP 200
Notifications	Low	Integrate Forgejo webhooks to send deployment status to a Matrix/Discord channel
Horizontal scaling	Low	Move from single server to Docker Swarm or Nomad when traffic demands it
Secrets manager	Low	Replace Forgejo Secrets with HashiCorp Vault or Infisical for rotation policies
Monitoring stack	Low	Add Prometheus + Grafana containers to the production compose for observability
Automatic rollback	Low	If the post-deploy health check fails, automatically roll back to the previous image tag

---

12. Decision Log

Decision	Chosen	Alternatives Considered	Rationale
Forgejo hosting	Self-hosted on Hetzner CX22 (~€5.49/mo)	Codeberg (free, FOSS only), Codey managed (~€39/mo), same-server co-hosting	Cheapest option with full control; separate server isolates CI builds from production; built-in container registry eliminates Docker Hub dependency
Server topology	Two CX22 servers + private network	Single server for everything, dedicated auction server	Isolation between CI and production workloads; either can be rebuilt independently; total cost ~€12/mo
CI/CD engine	Forgejo Actions	Woodpecker CI, Drone	Native integration with Forgejo, GitHub Actions syntax familiarity, built-in container registry
Container registry	Forgejo Packages	Harbor, Docker Hub	Zero additional infrastructure — ships with every Forgejo instance
IaC tool	OpenTofu	Pulumi, Ansible	Declarative HCL, excellent Hetzner provider, open-source fork of Terraform
Reverse proxy	Caddy	Nginx, Traefik	Automatic HTTPS with zero config, single binary, Caddyfile simplicity
Server provisioning	cloud-init	Ansible, Packer	No agent required, runs once at boot, perfect for immutable infrastructure
Deployment strategy	SSH + docker compose pull	Watchtower, Kubernetes	Simplest approach for a two-server deployment, easy to reason about
Persistent storage	Hetzner Volume	Local disk, NFS	Survives server rebuilds, attached to app server only