Server Infrastructure & Orchestration Architecture
How Trash Panda Hosting maximizes free community-funded game servers and low-jitter Mumble voice hosting per donated dollar—without burning 25% of community donations on orchestrator control-plane overhead.
Every MB of RAM saved on agents goes directly to community servers.
Why Not Raw Kubernetes on Day 1?
Why Kubernetes Is Tempting—And Where It Hurts
Heavy Tax on Small FleetsKubernetes (even K3s/Talos + Agones) excels at stateless HTTP microservices or massive studio matchmakers (500+ ephemeral battle-royale lobbies). For community-donation-funded persistent friend-group servers, it introduces four structural taxes:
-
The 1.5–2.5 GB/Node Control Plane & Agent Tax: Between
k3s-server/kubelet,containerd, CNI (Cilium/Calico), CoreDNS, metrics-server, and Agones controller pods, each node surrenders ~2.1 GB RAM before a single player joins. That equals 60+ Mumble servers or a full Minecraft server lost on every node. -
High-PPS UDP Jitter for Mumble Voice: Mumble transmits continuous 20ms Opus voice frames (
50 PPS × N listeners) over UDP plus TLS TCP control on the exact same port. Routing this throughkube-proxyiptables/IPVS or double-NAT conntrack tables causes micro-burst jitter and conntrack table exhaustion under UDP floods. -
Sticky Local NVMe World Saves Break Pod Portability: A 15 GB Minecraft
world/directory on local NVMe RAID-1 cannot live on high-latency network block storage (Ceph/Longhorn) without ruining chunk-save tick times. Once a pod is pinned to a node via a Local PVC, Kubernetes's marquee feature—automatic cross-node pod rescheduling—is neutralized anyway. - No Native First-Packet UDP Scale-to-Zero: Friend-group servers sit empty 85% of the week. K8s Ingress/Knative only scales HTTP on TCP. Waking a sleeping Minecraft/Valheim server on an arbitrary UDP query packet requires custom userspace/eBPF socket proxying regardless.
The Recommended Phased Architecture
4x–5x Server Density
Keep the central Go 1.25 + sqlc + PostgreSQL API as the single source of truth, and abstract node orchestration behind a clean NodeOrchestrator Go interface:
-
Phase 1–2 Winner (1 to 40 Nodes) — Option C: Bespoke Go Node Daemon (
raccoond): A single static 25 MB Go binary (inspired by Pterodactyl Wings & Fly.io'sflyd) communicating with the Go Control Plane over mTLS gRPC. Talks directly to the local Docker/containerd socket, binds host UDP/TCP ports directly, manages ZFS/Btrfs subvolume quotas, and runs a built-in UDP/TCP Wake-on-Packet Hibernation Proxy. -
Alternative Off-the-Shelf Runner-Up — Option B: HashiCorp Nomad: Single ~65 MB binary per node, native support for both
dockerand rawexec(cgroups v2isolated processes for ultra-denseumurmurdMumble processes without container overhead), static host port mapping, and zero CNI overlay tax. -
Phase 3 Escape Hatch (50+ Homogeneous Nodes) — Option A: K3s + Agones: Because the Go backend talks to an interface (
NodeOrchestrator), you can add anAgonesK8sDriverlater without rewriting a single SQL query or Angular component.
Option C: Bespoke Lightweight Go Agent (raccoond)
Why do almost all successful game server panels (Pterodactyl Wings, Pelican, PufferPanel) use a lightweight Go daemon per node talking to Docker/containerd instead of Kubernetes? Because game servers are stateful, port-pinned, console-streamed pets, not stateless HTTP cattle.
- Memory Footprint: ~28 MB RSS for
raccoond+ ~55 MB forcontainerd/Docker = ~83 MB total node overhead (96% less than K8s + Agones). - Native Wake-on-Packet Proxy: When a server hits 0 players for 15 minutes,
raccoondstops the container and binds a lightweight Go listener on its assigned port (e.g.,UDP/TCP :25565). The instant a player refreshes their server list or connects,raccoondwakes the container in ~2.5s. - Sub-Millisecond Mumble Voice: Spawns
murmurdwith--net=hostor directnftablesDSCPEF (0x2e)QoS tagging—zero userland proxy jitter. - Instant Log & RCON Streaming: Streams container stdout/stdin over gRPC back to the Angular Control Panel WebSocket/SSE console.
Node Spec & Wire Protocol (raccoond)
// raccoond per-node lifecycle (Go 1.25 + Docker Engine API + ZFS)
1. Control Plane assigns Server(id="srv_valheim_01", port=2456, node="us-east-1")
2. Control Plane -> mTLS gRPC -> raccoond.ProvisionServer()
3. raccoond creates ZFS dataset:
zfs create -o refquota=15G -o compression=lz4 tank/servers/srv_valheim_01
4. raccoond pulls OCI image & starts container with cgroups v2 limits:
CPUQuota: 200% (2 cores), MemoryMax: 4096MB, MemorySwapMax: 0
PortBindings: 2456/udp -> 2456/udp, 2457/udp -> 2457/udp
5. Idle Watchdog polls A2S_INFO / Minecraft SLP every 60s:
If 0 players for 15m -> container.Stop() -> WakeProxy.Listen(2456)
Option B: HashiCorp Nomad (+ Docker & Raw exec Drivers)
If you want battle-tested bin-packing and multi-node scheduling out of the box without writing cluster membership logic, HashiCorp Nomad is vastly better suited to game and voice hosting than Kubernetes.
- Memory Footprint: ~120–180 MB RAM per node. Single binary handles both server consensus (Raft) and client scheduling.
- First-Class Raw
execDriver for Mumble: Nomad doesn't force everything into a container. It can launchumurmurddirectly as an isolated Linux process usingchroot+cgroups v2, packing 250+ Mumble servers onto a single $6/mo VPS. - Trivial Static & Dynamic Host Ports: Nomad's
network { mode = "host" }natively maps UDP+TCP on the same port number withoutkube-proxyiptables chains. - Trade-off vs
raccoond: Nomad still requires a custom sidecar or node daemon to implement UDP Wake-on-Packet hibernation and ZFS subvolume quota creation.
Sample Nomad Job Spec (Mumble + Game Server)
job "tp-mumble-srv-884" {
datacenters = ["hetzner-fsn1", "ovh-bhs"]
type = "service"
group "voice" {
network {
mode = "host"
port "mumble" { static = 64738 } # Shared TCP + UDP port!
}
task "murmurd" {
driver = "exec" # Zero Docker overhead; raw cgroup isolation
config {
command = "/usr/sbin/murmurd"
args = ["-fg", "-ini", "${NOMAD_TASK_DIR}/murmur.ini"]
}
resources {
cpu = 150 # MHz
memory = 32 # MB RAM!
}
}
}
}
Option A: Kubernetes (K3s / Talos + Agones + Cilium)
If Trash Panda Hosting grows to 50+ bare-metal nodes or receives sponsored cloud credits (GKE/EKS), Kubernetes with Google's Agones framework becomes viable—provided you strip out default K8s networking traps.
- Mandatory Tuning if K8s is Used: You must disable
kube-proxyand run Cilium in kube-proxy-free eBPF HostPort mode so UDP voice/game packets bypass iptables NAT chains. - Agones GameServer CRD: Manages dynamic port allocation (
7000–8000/udp) and health checks via an injected sidecar SDK (~35 MB RAM overhead per game server pod). - Why Wait Until Phase 3: On a $45/mo 64 GB Hetzner node, K3s + Cilium + Agones + per-pod sidecars consume ~3.8 GB RAM across 25 pods and complicate ZFS local disk snapshotting.
Agones GameServer CRD Overhead Breakdown
apiVersion: "agones.dev/v1"
kind: GameServer
metadata:
name: "minecraft-survival-42"
spec:
ports:
- name: default
portPolicy: Static
hostPort: 25565
containerPort: 25565
protocol: TCPUDP # Requires K8s MixedProtocolLBService + Agones
template:
spec:
# Hidden cost: Agones injects agones-gameserver-sidecar (~32MB RAM)
# into EVERY pod, which doubles the RAM cost of a 25MB Mumble server!
containers:
- name: paper-mc
image: itzg/minecraft-server:java21
Why Game & Voice Servers Break Cloud-Native Assumptions
1. Mumble Voice (murmurd / umurmurd)
15–40 MB RAM
Mumble is an engineering marvel: a 50-slot encrypted Opus voice server written in C/C++ consumes less RAM than a single Chrome tab, but has strict real-time network constraints:
- Ultra-Dense Memory Profile:
umurmurduses ~8–15 MB RAM; fullmurmurdwith SQLite channel ACLs uses ~20–40 MB RAM. - 50 PPS Isochronous UDP Stream: Every active speaker sends 1 Opus packet every 20ms (50 packets/sec). In a 10-person raid with 3 people talking, the server fans out
3 × 9 × 50 = 1,350 PPS. - Dual TCP+UDP Same-Port Binding: Mumble binds TCP (TLS 1.3 control/auth/crypt-setup) and UDP (OCB2-AES128 voice frames) on the exact same port (e.g.,
64738). - Zero NAT Re-mapping Tolerance: If an ingress proxy or full-cone NAT rewrites source UDP ports mid-session, Mumble drops back to TCP tunneling, causing 150ms+ voice lag spikes.
2. Game Servers (MC, Valheim, CS2)
2–8 GB RAM + High IPCUnlike stateless web workers that scale horizontally across slow vCPUs, game servers run a monolithic deterministic simulation loop (20 TPS for Minecraft, 64/128 tick for Source, 60 UPS for Factorio):
- Single-Thread Turbo Sensitivity: World entity tick loops are predominantly single-threaded. A Ryzen 9 7950X / i9-13900K core at 5.4 GHz outperforms 4x EPYC cloud vCPUs at 2.4 GHz.
- Stateful Local NVMe I/O Bursts: Every 5 minutes, region/world autosaves flush 50–400 MB synchronously via
fsync(). Networked block storage (EBS/Ceph) causes "Can't keep up! Did the system time change?" tick stalls. - Protocol Diversity: Minecraft Java uses TCP (
25565) + optional UDP Simple Voice Chat (24454); Valheim uses 2 consecutive UDP ports (2456-2457); Source uses TCP RCON + UDP game on the same port.
3. The Hibernation Multiplier
4x–5x Donation LeverageIn commercial paid hosting, users pay $15/mo for an empty server burning electricity 24/7. In donation-funded free hosting, keeping idle JVMs resident in RAM bankrupts the community pool.
- The 85% Idle Law: A friend group's Minecraft or Valheim server is actively played ~20–26 hours per week (evenings & weekends) and sits at
0/10 playersfor the remaining 142+ hours/week. - Scale-to-Zero Disk Residency: A hibernating server costs 0 MB of RAM and only ~4 GB of compressed NVMe disk (which costs ~$0.03/mo on Hetzner NVMe drives).
- Seamless Wake-on-Packet: With
raccoondholding the port while the container sleeps, friends don't even have to log into the web panel to start the server—connecting from the game client wakes it automatically!
Deep 3-Way Comparison: K8s vs. Nomad vs. Bespoke raccoond
| Engineering Dimension | Option 1: Kubernetes K3s / Talos + Agones + Cilium HostPort |
Option 2: HashiCorp Nomad Nomad + Docker & Raw exec Drivers |
Option 3: Bespoke Go Agent (raccoond) ★gRPC Daemon + Docker API + UDP Wake Proxy |
|---|---|---|---|
| 1. Node RAM Overhead Control plane + agent + per-server sidecar tax |
1.8 GB – 2.5 GB / node Plus ~32 MB per pod for the injected agones-gameserver-sidecar. Doubles the RAM cost of every Mumble server.
|
120 MB – 180 MB / node Single Go binary. Zero per-server sidecar tax. Raw exec driver skips container shim overhead for Mumble.
|
~65 MB – 85 MB / node Static Go binary (~25 MB RSS) + containerd. Zero sidecars. 99% of node RAM goes directly to user servers.
|
| 2. UDP Networking & Mumble Jitter 20ms Opus audio frames & TCP+UDP shared ports |
High Friction Default kube-proxy iptables adds conntrack latency & UDP drops. Requires Cilium eBPF Socket-LB + MixedProtocolLBService.
|
Native Host Mode Direct host port binding for TCP+UDP on the same port number. Zero overlay CNI required for game/voice traffic. |
Zero-Copy Host Binding Direct host socket binding + automatic nftables DSCP EF prioritization for Mumble UDP voice packets.
|
| 3. Scale-to-Zero / Wake-on-Packet Pausing empty servers & waking on UDP/TCP packet |
Not Native for UDP Agones assumes pre-warmed active fleets. Waking a terminated pod on a raw UDP query packet requires custom external proxy infrastructure anyway. |
Requires Companion Proxy Can scale job count to 0 via API, but requires a custom port-holding proxy daemon on each node to detect incoming UDP packets. |
Built Into Daemon Coreraccoond natively unbinds container, binds UDP/TCP port in-process, inspects handshake/A2S packet, and starts container in ~2.5s.
|
| 4. Stateful World-Save Storage NVMe speed, quotas & instant rollback snapshots |
Complex CSI Driverslocal-path-provisioner lacks hard disk quotas out of the box; TopoLVM/ZFS-LocalPV adds heavy CSI controller pods.
|
Host Volumes Uses Nomad Host Volumes; quota enforcement and pre-provision directory setup require pre-start hooks. |
Direct ZFS / Btrfs Subvolumes Creates a 15 GB quota-capped subvolume in 8ms ( zfs create -o refquota=15G) with zero-cost copy-on-write backups.
|
| 5. Bare-Metal Heterogeneity Mixing Hetzner auction boxes, OVH Eco & mini-PCs |
Poor on Edge / Low-RAM Nodes etcd requires low-latency disk quorum; running K8s across WAN links (US + EU + donated ARM boxes) is brittle. |
Excellent Federation Nomad handles multi-region WAN federation and mixed x86_64 / ARM64 nodes cleanly. |
Runs Anywhere Linux Runs Stateless node agent connects outbound via mTLS gRPC to the central Go API. Can turn a $5 VPS into a 150-server Mumble node in 10 seconds. |
| 6. Operational Complexity (Solo Admin) 2 AM debugging, upgrades & failure blast radius |
High Cognitive Load CNI upgrades, expired cluster certs, stuck Terminating pods, CRD version skew, and etcd split-brain. |
Low Cognitive Loadsystemctl restart nomad. Simple HCL job files. If control plane restarts, running game servers never drop a packet.
|
Minimal (~1,200 LOC Go)scp raccoond node:/usr/local/bin/. Full codebase lives in the same Go monorepo as the backend API.
|
| 7. Security & Multi-Tenant Isolation Containing untrusted community mods & plugins |
Strong (PodSecurity + RuntimeClass) Native support for gVisor ( runsc) RuntimeClass and Cilium NetworkPolicies.
|
Strong (cgroups v2 + Docker/gVisor) Supports rootless Docker, runsc runtime, and Linux capability dropping.
|
Tailored Game Sandbox Enforces cgroups v2, no-new-privileges, read-only rootfs, optional runsc (gVisor), and per-server nftables anti-DDoS rules.
|
Interactive Architecture & Wake-on-Packet Diagrams
Diagram 1: End-to-End Trash Panda Hosting Topology
Separation of the Central Control Plane (Astro + Angular + Go API + Postgres) from Bare-Metal Voice & Game Data Plane Nodes.
Diagram 2: The "Wake-on-Packet" (Scale-to-Zero) State Machine
How raccoond achieves 3x–4.5x game server density on donated hardware without asking players to manually start servers in the web UI.
Donation Capacity & "Control Plane Tax" Calculator
Adjust monthly community donations, bare-metal node size, and the idle hibernation overcommit ratio below to see how many Mumble Voice Servers (32 MB avg) and Game Servers (3.5 GB avg) Trash Panda Hosting can sustain under each architecture.
Community Fleet Inputs
Based on Hetzner Dedicated / Auction bare-metal pricing (~$45/mo for 64 GB NVMe, ~$30/mo for 32 GB, ~$82/mo for 128 GB).
Option C: Bespoke raccoond Go Daemon + Wake-on-Packet Proxy ★
~80 MB agent overhead/node · Native UDP/TCP Wake-on-Packet hibernation supported
Option B: HashiCorp Nomad (Docker + Raw exec Drivers)
~160 MB overhead/node · Requires external companion proxy for UDP scale-to-zero (1.5x partial hibernation without custom proxy)
Option A: Kubernetes (K3s + Agones + Cilium + Sidecars)
~2.2 GB node overhead + 32 MB Agones SDK sidecar per pod · Always-on pods (1.0x) unless custom UDP proxy is builtSecurity & Anti-Abuse Architecture for Free Hosting
Any platform offering free compute is an immediate target for crypto-miners (XMRig disguised as java), outbound UDP reflection/DDoS scripts, zip/disk bombs, and noisy-neighbor CPU starvation. Trash Panda Hosting enforces four kernel-level guardrails on every node:
1. cgroups v2 CPU Burst Caps & Miner Watchdog
-
Hard CPU & PID Quotas: Every container runs inside a dedicated
cgroups v2slice withcpu.max(e.g.200000 100000= 2.0 cores max),memory.swap.max = 0(prevents thrashing NVMe swap), andpids.max = 512(blocks fork bombs). -
Zero-Player CPU Anomaly Detector: Crypto-miners peg CPU at 100% 24/7 while reporting 0 connected players. If
raccoondobservesCPU > 85%for 10 consecutive minutes while game-protocol query reports0 players(after initial world generation window), the container is automatically paused and flagged for admin review.
2. nftables Egress Allowlists & Anti-Amplification
-
Default-Deny Outbound Arbitrary Ports: Game servers only need outbound HTTPS (
:443) to Mojang/Steam/Modrinth auth servers and DNS (:53).raccoondinstalls annftableschain per container blocking outbound connections to RFC1918 private subnets, metadata IPs (169.254.169.254), and Stratum mining ports (3333, 4444, 5555, 14444). -
UDP Egress PPS Rate-Limiting: Prevents compromised game plugins from launching outbound UDP floods (
limit rate over 4000/second dropper container IP/port).
3. ZFS / Btrfs Subvolume Quotas & NVMe IOPS Caps
-
Instant Hard Disk Quotas: Standard ext4 Docker overlayfs lets a malicious plugin write a 500 GB log file and crash the entire host node in 90 seconds. Each server gets an isolated ZFS dataset (
tank/servers/<uuid>) withrefquota=15Gandcompression=zstd-3. -
Block I/O Fairness:
cgroups v2io.maxcaps per-container write bandwidth atrbps=250M wbps=150M riops=4000 wiops=2000so one server's world-gen cannot stall Mumble audio or sibling ticks.
4. Rootless Execution & gVisor (runsc) Option
-
Unprivileged UID Mapping: All server processes run as non-root UID
10000+Nvia user namespaces, with read-only root filesystems,cap-drop=ALL, andsecurity-opt=no-new-privileges. -
gVisor Sandbox for Custom Modpacks: When users upload untrusted native `.so` plugins,
raccoondcan selectRuntime: "runsc"(gVisor) to intercept syscalls in userspace.
Proposed Implementation Blueprint: NodeOrchestrator Interface
By defining this single interface in backend/internal/orchestrator/orchestrator.go, Trash Panda Hosting's HTTP handlers, sqlc PostgreSQL queries, and Angular Control Panel remain 100% decoupled from whether the underlying node runs raccoond, HashiCorp Nomad, or Kubernetes + Agones:
package orchestrator
import (
"context"
"io"
"time"
)
// ServerSpec defines the desired resource and network allocation on a target bare-metal node.
type ServerSpec struct {
ServerID string `json:"serverId"`
NodeID string `json:"nodeId"`
Kind string `json:"kind"` // "game" | "voice"
Template string `json:"template"` // "minecraft-paper", "valheim", "mumble", etc.
Image string `json:"image"` // OCI image or binary target
HostIP string `json:"hostIp"`
Port int `json:"port"` // Shared TCP+UDP primary port
MemoryMB int `json:"memoryMb"` // Hard cgroups v2 memory.max
CPUMillicores int `json:"cpuMillicores"` // e.g. 2000 = 2.0 cores
DiskQuotaMB int `json:"diskQuotaMb"` // ZFS/Btrfs subvolume refquota
MaxPlayers int `json:"maxPlayers"`
HibernationEnabled bool `json:"hibernationEnabled"` // Enables 15m 0-player Wake-on-Packet proxy
Env map[string]string `json:"env"`
}
// RuntimeStatus reports live telemetry from the node agent back to the Go Control Plane.
type RuntimeStatus struct {
State string `json:"state"` // "running" | "stopped" | "hibernating" | "provisioning"
CurrentPlayers int `json:"currentPlayers"` // Sampled via SLP / A2S_INFO / Murmur RPC
MemoryUsedMB int `json:"memoryUsedMb"`
CPUPercent float64 `json:"cpuPercent"`
DiskUsedMB int `json:"diskUsedMb"`
LastActiveAt time.Time `json:"lastActiveAt"`
}
// NodeOrchestrator abstracts raccoond (mTLS gRPC), HashiCorp Nomad (HTTP API), and Kubernetes (Agones CRD).
type NodeOrchestrator interface {
// ProvisionServer creates the ZFS/Btrfs quota volume, pulls the image, and starts or registers the server.
ProvisionServer(ctx context.Context, spec ServerSpec) (*RuntimeStatus, error)
// StartServer wakes a hibernating or stopped server and hands the host UDP/TCP port to the container.
StartServer(ctx context.Context, nodeID, serverID string) (*RuntimeStatus, error)
// StopServer sends a graceful save command (RCON/SIGTERM) and stops the container without binding the wake proxy.
StopServer(ctx context.Context, nodeID, serverID string) (*RuntimeStatus, error)
// HibernateServer stops the container, snapshots the disk, and binds raccoond's UDP/TCP Wake-on-Packet listener.
HibernateServer(ctx context.Context, nodeID, serverID string, port int) (*RuntimeStatus, error)
// DeprovisionServer removes the container, unbinds ports, and destroys the ZFS/Btrfs subvolume.
DeprovisionServer(ctx context.Context, nodeID, serverID string) error
// StreamConsole attaches to live stdout/stderr and accepts RCON/stdin commands for the Angular panel.
StreamConsole(ctx context.Context, nodeID, serverID string, stdin io.Reader) (io.ReadCloser, error)
}
// Driver implementations selectable per-node via Node.Orchestrator ("raccoond" | "nomad" | "agones-k8s"):
// - RaccoonDaemonDriver: mTLS gRPC client to raccoond on :9443 (Default for Phase 1–2)
// - NomadDriver: Nomad HTTP v1 Jobs API client (docker + raw exec task drivers)
// - AgonesK8sDriver: client-go dynamic client managing agones.dev/v1 GameServer CRDs