Complete guide to setting up K3s, Docker, and a private container
registry on the TechnoCore Proxmox infrastructure.
See also:
resource.notes/howto/howto_docker_build.md — building Axion Docker imagesresource.notes/howto/deployment_architecture.md — architecture diagrams (local + cloud)resource.notes/howto/cloudflare-cdn-setup.md — DNS and CDN configuration10.0.10.18 (node: hypervisor)local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst)vmbr1 bridge on 10.0.10.0/24, gateway 10.0.10.254anchor.technocore.co.za:51820 (subnet 10.100.0.0/24)Via Proxmox API (authenticated as root@pam):
# Authenticate
TICKET=$(curl -sk -d 'username=root@pam&password=<password>' \
"https://10.0.10.18:8006/api2/json/access/ticket" | \
python3 -c "import sys,json; d=json.loads(sys.stdin.read(),strict=False); print(d['data']['ticket'])")
CSRF=$(curl -sk -d 'username=root@pam&password=<password>' \
"https://10.0.10.18:8006/api2/json/access/ticket" | \
python3 -c "import sys,json; d=json.loads(sys.stdin.read(),strict=False); print(d['data']['CSRFPreventionToken'])")
# Create LXC container
curl -sk -b "PVEAuthCookie=$TICKET" -H "CSRFPreventionToken: $CSRF" \
-X POST "https://10.0.10.18:8006/api2/json/nodes/hypervisor/lxc" \
--data-urlencode "vmid=136" \
--data-urlencode "hostname=registry" \
--data-urlencode "ostemplate=local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst" \
--data-urlencode "storage=ztechno" \
--data-urlencode "rootfs=ztechno:20" \
--data-urlencode "cores=2" \
--data-urlencode "memory=2048" \
--data-urlencode "swap=512" \
--data-urlencode "net0=name=eth0,bridge=vmbr1,firewall=1,gw=10.0.10.254,ip=10.0.10.5/24,type=veth" \
--data-urlencode "password=<password>" \
--data-urlencode "unprivileged=0" \
--data-urlencode "features=nesting=1,keyctl=1" \
--data-urlencode "start=1"
Critical notes:
Bridge name: Use vmbr1, not vmbr0. Check existing containers
with pct exec <vmid> -- cat /etc/network/interfaces or the Proxmox
API config endpoint to confirm the correct bridge.
Privileged container: unprivileged=0 is required for Docker
inside LXC. Docker needs access to cgroups and device nodes that
unprivileged containers cannot provide.
Nesting: features=nesting=1,keyctl=1 is required for Docker
to function inside LXC. Without nesting, container runtimes
cannot create their own namespaces. Without keyctl, some kernel
security features break.
Static IP: Set ip=10.0.10.5/24 in the net0 parameter. If
the container was created with DHCP, reconfigure via the API:
curl -sk -b "PVEAuthCookie=$TICKET" -H "CSRFPreventionToken: $CSRF" \
-X PUT "https://10.0.10.18:8006/api2/json/nodes/hypervisor/lxc/136/config" \
--data-urlencode "net0=name=eth0,bridge=vmbr1,firewall=1,gw=10.0.10.254,ip=10.0.10.5/24,type=veth"
The container must be stopped before network config changes take effect.
The initial 20GB is too small for a registry. Resize to 100GB:
curl -sk -b "PVEAuthCookie=$TICKET" -H "CSRFPreventionToken: $CSRF" \
-X PUT "https://10.0.10.18:8006/api2/json/nodes/hypervisor/lxc/136/resize" \
--data-urlencode "disk=rootfs" \
--data-urlencode "size=100G"
Problem: New LXC containers on this Proxmox host inherit a Netbird
DNS resolver (100.79.37.155) from the PVE-managed /etc/resolv.conf.
This resolver cannot resolve public domains (deb.debian.org etc.),
which breaks apt-get.
Fix: Write resolv.conf directly inside the container:
# From inside the container (via SSH or pct exec)
echo 'nameserver 8.8.8.8' > /etc/resolv.conf
echo 'nameserver 1.1.1.1' >> /etc/resolv.conf
You can also set nameservers via the Proxmox API, but this only takes
effect on container restart:
curl -sk -b "PVEAuthCookie=$TICKET" -H "CSRFPreventionToken: $CSRF" \
-X PUT "https://10.0.10.18:8006/api2/json/nodes/hypervisor/lxc/136/config" \
--data-urlencode "nameserver=8.8.8.8 1.1.1.1"
Verify DNS works before proceeding:
ping -c1 deb.debian.org
Password-based SSH may not work immediately on new Debian 13 LXC
containers. Push SSH keys via the Proxmox host using pct push:
# On the Proxmox host
pct exec 136 -- mkdir -p /root/.ssh
pct exec 136 -- chmod 700 /root/.ssh
pct push 136 /tmp/my_pubkey.txt /root/.ssh/authorized_keys
pct exec 136 -- chmod 600 /root/.ssh/authorized_keys
Note: The Proxmox API ssh-public-keys parameter only works at
container creation time, not on existing containers.
apt-get update -qq
apt-get install -y -qq ca-certificates curl gnupg
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg \
-o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] \
https://download.docker.com/linux/debian trixie stable" \
> /etc/apt/sources.list.d/docker.list
apt-get update -qq
apt-get install -y -qq docker-ce docker-ce-cli containerd.io
systemctl enable docker
systemctl start docker
Verify: docker version --format '{{.Server.Version}}'
docker run -d \
--name registry \
--restart=always \
-p 5000:5000 \
-v /var/lib/registry:/var/lib/registry \
registry:2
Verify: curl -s https://registry.technocore.co.za/v2/_catalog should return
{"repositories":[]}.
| Property | Value |
|---|---|
| LXC VMID | 136 |
| Hostname | registry |
| LAN IP | 10.0.10.5 |
| WireGuard IP | 10.100.0.5 |
| Registry port | 5000 |
| Disk | 100GB (ztechno ZFS) |
| OS | Debian 13 (trixie) |
| Docker | 29.4.1 |
apt-get install -y -qq wireguard-tools
wg genkey | tee /etc/wireguard/private.key | wg pubkey > /etc/wireguard/public.key
chmod 600 /etc/wireguard/private.key
PRIVATE_KEY=$(cat /etc/wireguard/private.key)
cat > /etc/wireguard/wg0.conf << EOF
[Interface]
Address = 10.100.0.5/24
PrivateKey = ${PRIVATE_KEY}
[Peer]
# Anchor server (WG hub)
PublicKey = lk8ch/1EYahIi6uQf9WAyj+zxbiGfdGQG25OTl9OPyQ=
Endpoint = anchor.technocore.co.za:51820
AllowedIPs = 10.100.0.0/24
PersistentKeepalive = 25
[Peer]
# Proxmox hypervisor (direct local peer)
PublicKey = uSKXOE5vabNlWQAltCHlWgkf+2ItH05z9bwg0ek7T0Y=
AllowedIPs = 10.100.0.18/32
Endpoint = 10.0.10.18:34490
PersistentKeepalive = 25
EOF
chmod 600 /etc/wireguard/wg0.conf
systemctl enable wg-quick@wg0
systemctl start wg-quick@wg0
This must be done on the anchor server (anchor.technocore.co.za):
wg set wg0 peer <REGISTRY_PUBLIC_KEY> allowed-ips 10.100.0.5/32
wg-quick save wg0
Important: The anchor server at anchor.technocore.co.za requires
SSH key authentication. No password auth is accepted. Access via AWS
Console Session Manager if no authorised key is available locally.
# On Proxmox host
wg set wg0 peer <REGISTRY_PUBLIC_KEY> \
allowed-ips 10.100.0.5/32 \
endpoint 10.0.10.5:<REGISTRY_WG_PORT>
wg-quick save wg0
# From the registry container
wg show wg0 # Both peers should show recent handshake
ping -c1 10.100.0.18 # Proxmox via WG
ping -c1 10.100.0.1 # Anchor via WG
curl -sfL https://get.k3s.io | \
INSTALL_K3S_EXEC="--write-kubeconfig-mode 644 --disable traefik --docker" sh -
Flags explained:
--write-kubeconfig-mode 644: Makes kubeconfig readable without--disable traefik: We use Caddy as the reverse proxy, not--docker: Use Docker as the container runtime instead of containerd.docker build images to be available to k3smkdir -p ~/.kube
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
chmod 600 ~/.kube/config
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
kubectl get nodes # Should show Ready
kubectl get pods -A # Should show coredns, metrics-server, local-path-provisioner
helm version --short # Should show v3.x
The registry is available at registry.technocore.co.za (HTTPS via
Caddy on anchor) and 10.0.10.5:5000 (HTTP, LAN only).
Create /etc/rancher/k3s/registries.yaml (or generate via
python factory.deploy/ObjBuild.py scaffold homechoice):
mirrors:
"registry.technocore.co.za":
endpoint:
- "https://registry.technocore.co.za"
"10.0.10.5:5000":
endpoint:
- "http://10.0.10.5:5000"
Then restart k3s:
sudo systemctl restart k3s
For Docker daemon (needed for LAN docker push):
Add to /etc/docker/daemon.json:
{
"insecure-registries": ["10.0.10.5:5000"]
}
Then restart Docker:
sudo systemctl restart docker
Warning: Restarting Docker while a Docker build is running will
kill the build with rpc error: code = Unavailable desc = error reading from server: EOF. Always check for running builds with
ps aux | grep "docker build" before restarting the daemon.
If you need to remove k3s:
/usr/local/bin/k3s-uninstall.sh
The Axion Dockerfile is a 3-stage build:
uvfactory.* modules via ObjCompile.so filesPer-service targets (report, webhook, web, etc.) extend the runtime
stage with a fixed AXION_SERVICE and default CMD.
docker build \
--progress=plain \
--build-arg AXION_PACKAGE=homechoice \
--target report \
-t registry.technocore.co.za/axion/homechoice-report:latest \
-f resource.docker/dockerfile/homechoice.dockerfile .
Build times (approximate, cold cache):
| Stage | Time |
|---|---|
| builder (venv + pip) | 3-5 min |
| compiler (Cython) | 5-10 min |
| runtime (Playwright) | 1-2 min |
| Total | 10-17 min |
Cached rebuilds (code-only changes) skip builder and runtime stages,
taking 5-10 minutes for the Cython compile.
The registry LXC (136) doubles as a remote buildx builder with 24
cores and 64GB RAM. It supports ARM64 via QEMU emulation.
# ARM64 via the pipeline
python factory.deploy/ObjBuild.py pipeline homechoice --platform linux/arm64
# Both architectures (multi-arch manifest)
python factory.deploy/ObjBuild.py pipeline homechoice \
--platform linux/amd64,linux/arm64
When --platform is set, buildx pushes directly to the registry
(--push). ARM builds take 2-4x longer than native amd64 due to
QEMU emulation.
Packages without ARM wheels are listed in
resource.config/requirements-arm-exclude.txt and automatically
stripped on non-amd64 targets.
docker push registry.technocore.co.za/axion/homechoice-report:latest
Verify:
curl -s https://registry.technocore.co.za/v2/_catalog
# {"repositories":["axion/homechoice","axion/homechoice-report"]}
curl -s https://registry.technocore.co.za/v2/axion/homechoice-report/tags/list
# {"name":"axion/homechoice-report","tags":["latest"]}
Generate the Helm chart scaffold:
AXION_DEPLOYMENT_SUBSTRATE=k3s python factory.deploy/ObjBuild.py scaffold homechoice
This creates resource.docker/helm/homechoice/ with:
Chart.yaml — chart metadatavalues.yaml — service topology, registry, replicastemplates/deployment.yaml — one Deployment per servicetemplates/service.yaml — one ClusterIP Service per servicetemplates/configmap-caddy.yaml — Caddy reverse proxy configDeploy:
helm upgrade --install homechoice resource.docker/helm/homechoice/ \
--namespace homechoice-dev \
--create-namespace
The base.deployment section in config.yaml drives the entire
deployment topology:
base:
deployment:
substrate: docker # K3S | DOCKER | SUPERVISOR | LXC
registry:
host: "10.0.10.5"
port: 5000
domain: "$dns$"
caddy:
email: "admin@technocore.co.za"
Per-package overrides:
homechoice:
deployment:
substrate: k3s
namespace: homechoice-dev
Override at runtime via environment variable:
AXION_DEPLOYMENT_SUBSTRATE=k3s python factory.deploy/ObjBuild.py scaffold homechoice
The base.services section defines instance counts and ports:
services:
report:
port_start: 9400
instances: 5
webhook:
port_start: 9500
instances: 10
How instances is interpreted per substrate:
| Substrate | Meaning |
|---|---|
| K3S | Kubernetes replicas: in Deployment spec |
| DOCKER | N containers, each on port_start + N |
| SUPERVISOR | N [program:] blocks on consecutive ports |
| LXC | N LXC containers with per-container port |
# Docker (default)
python factory.deploy/ObjBuild.py scaffold homechoice
# Generates: docker-compose.yaml + Caddyfile
# K3s
AXION_DEPLOYMENT_SUBSTRATE=k3s python factory.deploy/ObjBuild.py scaffold homechoice
# Generates: Helm chart + Caddyfile
# Supervisor
AXION_DEPLOYMENT_SUBSTRATE=supervisor python factory.deploy/ObjBuild.py scaffold homechoice
# Generates: supervisor.conf + Caddyfile
# LXC
AXION_DEPLOYMENT_SUBSTRATE=lxc python factory.deploy/ObjBuild.py scaffold homechoice
# Generates: containers.yaml + Caddyfile
# Password auth (returns ticket + CSRF token)
curl -sk -d 'username=root@pam&password=<pw>' \
"https://10.0.10.18:8006/api2/json/access/ticket"
# Token auth (no ticket needed)
curl -sk -H "Authorization: PVEAPIToken=axionapi@pve!mastertoken=<uuid>" \
"https://10.0.10.18:8006/api2/json/..."
Note: The API token axionapi@pve!mastertoken has limited
permissions. Use root@pam password auth for container creation
and management.
# List containers
curl -sk -b "PVEAuthCookie=$TICKET" \
"https://10.0.10.18:8006/api2/json/nodes/hypervisor/lxc"
# Container status
curl -sk -b "PVEAuthCookie=$TICKET" \
"https://10.0.10.18:8006/api2/json/nodes/hypervisor/lxc/136/status/current"
# Start/stop
curl -sk -b "PVEAuthCookie=$TICKET" -H "CSRFPreventionToken: $CSRF" \
-X POST "https://10.0.10.18:8006/api2/json/nodes/hypervisor/lxc/136/status/start"
# Execute command (NOT available on PVE 9.1.9 for LXC)
# Use pct exec via SSH to Proxmox host instead
net0 parameter format: Must use key=value pairs joined byinvalid format - value without key errors. Correct: name=eth0,bridge=vmbr1,ip=10.0.10.5/24,gw=10.0.10.254,type=vethfeatures parameter: Same format requirement. Correct:nesting=1,keyctl=1--data-urlencode: Always use --data-urlencode instead of-d for Proxmox API calls. Parameters containing =, /, or-d.ssh-public-keys: Only works during container creation, notpct push instead.pct exec: Available via SSH to the Proxmox host. The RESTPOST .../exec) is not implemented in PVE 9.1.9Internet
│
├── anchor.technocore.co.za (13.244.252.132)
│ WG: 10.100.0.1
│ WG port: 51820
│
╰── WireGuard mesh (10.100.0.0/24)
│
├── 10.100.0.18 Proxmox hypervisor (10.0.10.18)
├── 10.100.0.5 Registry LXC 136 (10.0.10.5)
│
╰── LAN 10.0.10.0/24 (bridge vmbr1, gw .254)
│
├── .3 keycloak (LXC 112)
├── .5 registry (LXC 136) ← Docker registry:2 on :5000
├── .11 infisical (LXC 128)
├── .14 neo4j (LXC 107)
├── .16 hivemq (LXC 111)
├── .17 portainer (LXC 106)
├── .18 proxmox hypervisor
├── .20 mariadb (LXC 100)
├── .21 mariadblts (LXC 118)
├── .22 mariadbsim (LXC 119)
├── .23 mailrelay (LXC 108)
├── .24 influxdb (LXC 104)
├── .25 asterix (LXC 127)
├── .26 rabbitmq (LXC 103)
├── .29 sftpgo (LXC 113)
├── .32 monitor (LXC 124)
├── .33 traccar (LXC 134)
├── .35 Windev (VM 125)
├── .48 ollama (LXC 135)
├── .60 gekkoridge (LXC 133)
├── .61 samdev (LXC 102)
├── .62 fullhouse (LXC 121)
├── .63 tanyadev (LXC 114)
├── .64 finchoice (LXC 129)
├── .65 homechoice (LXC 126)
├── .66 owendev (LXC 105)
├── .68 pipeline (LXC 109)
├── .70 rewards (LXC 115)
└── .71 axiondev (LXC 120)
K3s VMs:
VM 149 k3s-template (stopped)
VM 150 k3s-server (running)
VM 151 k3s-agent (running)
Portainer CE runs on LXC 106 (10.0.10.17:9443). Harbor is also
co-located on this container.
Portainer is not directly accessible from outside the LXC — Docker
proxy binds to 0.0.0.0:9443 inside the container but the LXC
firewall or network config may restrict external access.
From inside the container (via pct exec 106 on Proxmox):
curl -sk https://localhost:9443/api/status
The Portainer API uses capitalised field names (Username, Password,
not username, password). Using lowercase returns HTTP 422
Invalid credentials without a clear error message.
import json, urllib.request, ssl
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
base = "https://localhost:9443/api"
# Authenticate
auth = json.dumps({
"Username": "admin",
"Password": "<password>"
}).encode()
req = urllib.request.Request(
f"{base}/auth",
data=auth,
headers={"Content-Type": "application/json"},
)
jwt = json.loads(
urllib.request.urlopen(req, context=ctx).read()
)["jwt"]
headers = {
"Authorization": f"Bearer {jwt}",
"Content-Type": "application/json",
}
# Add registry (Type 3 = custom registry)
reg = json.dumps({
"Name": "axion-registry",
"Type": 3,
"URL": "registry.technocore.co.za",
"Authentication": False,
}).encode()
req = urllib.request.Request(
f"{base}/registries",
data=reg,
headers=headers,
method="POST",
)
result = json.loads(
urllib.request.urlopen(req, context=ctx).read()
)
print(f"Registry added: ID={result['Id']}")
Username, Password,Name, Type, URL). Using lowercase causes 422 errors thatdocker stop portainer
docker run --rm -v portainer_data:/data portainer/helper-reset-password
docker start portainer
The Axion build pipeline in factory.deploy/ObjBuild.py supports
four deployment substrates. The substrate determines how service
instances are realised from the base.services config.
factory.deploy/extend.runner/
ObjBuildRunnerBase.py — abstract interface
ObjBuildRunnerK3s.py — Helm chart + k8s Caddy
ObjBuildRunnerDocker.py — docker-compose + port Caddy
ObjBuildRunnerSupervisor.py — supervisor.conf + port Caddy
ObjBuildRunnerLxc.py — LXC profiles + container Caddy
factory.core/extend.environment/
ObjRunnerBase.py — abstract interface
ObjRunnerK3s.py — kubectl start/stop/restart
ObjRunnerDocker.py — docker start/stop/restart
ObjRunnerSupervisor.py — supervisorctl start/stop/restart
ObjRunnerLxc.py — lxc start/stop/restart
Selected by environment.runner in config.yaml (runtime ops) and
deployment.substrate in config.yaml (build/deploy ops).
ObjBuild.pipeline(package):
1. Validate config.yaml
2. Check PEM keys exist
3. Cython compile → .so files
4. Docker build (always — all substrates need compiled images)
5. Docker push to registry
6. Build runner: topology → scaffold → deploy
├── K3S: Helm chart → helm install
├── DOCKER: docker-compose → docker compose up
├── SUPERVISOR: supervisor.conf → supervisorctl reload
└── LXC: containers.yaml → lxc launch
If you modify /etc/docker/daemon.json (e.g. to add insecure
registries) and restart Docker while a build is running, the build
will fail with:
ERROR: failed to build: rpc error: code = Unavailable
desc = error reading from server: EOF
Always check for running builds before restarting Docker:
ps aux | grep "docker build" | grep -v grep
The cached layers survive the restart — just re-run the build command
and it will resume from the last completed stage.