Building a Cloud-Agnostic Kubernetes Platform for Multi-Tenant Applications
I wanted a repeatable Kubernetes environment where I could work on the harder parts of a multi-tenant platform: networking, storage isolation, tenant provisioning, and recovery.
I also wanted to avoid designing everything around one cloud provider from day one. My starting point was a local KVM/libvirt lab. The longer-term plan is to provision infrastructure through provider APIs and move the machine configuration and application deployment into Ansible, Terraform, and Helm.
This post covers Cluster Bootstrap v0.1.0, the first tested baseline. It is a working infrastructure foundation, not yet a production-ready multi-tenant platform.
What the first version actually builds
The bootstrap script runs on a Linux KVM host and provisions:
- A separate management VM running Rancher in Docker.
- Two RKE2 server VMs and three RKE2 worker VMs.
- A libvirt NAT network (
192.168.100.0/24). - Cilium as the CNI, with WireGuard encryption and kube-proxy replacement.
- Cilium Envoy Ingress on NodePorts
30080(HTTP) and30443(HTTPS). - Longhorn with an encrypted-by-default StorageClass.
- A small controller that generates a distinct encryption Secret for each encrypted PVC.
The diagram separates the Rancher management VM from the downstream RKE2 nodes. The network and storage boxes represent services running inside the downstream cluster, not additional VMs.
Rancher is intentionally outside the downstream RKE2 cluster. It runs in its own VM, while the five other VMs form the managed cluster.
This is a lab deployment. The baseline deliberately uses some conveniences, including self-signed TLS and relaxed SSH host-key checking, that should not be carried into production.
Why start with Bash and KVM?
The first goal was to prove the sequence of operations before introducing more abstraction.
The script creates or reuses VM disks, sets up cloud-init, starts the management and RKE2 machines, waits for services, imports the cluster into Rancher, and configures storage. It also handles host-side details that are easy to overlook when focusing only on Kubernetes YAML: libvirt network definitions, VM image permissions, orphaned disks, and the order in which services become available.
I chose a simple Bash bootstrap first because I wanted to see the full process and its failure modes. Terraform and Ansible are planned, but neither is part of this baseline.
Bootstrap order and dependencies
The sequence is based on the order in the frozen script: management VM and Rancher first, then downstream RKE2 VMs; Cilium configuration is written before the first RKE2 server starts. After all five nodes become Ready, the script imports the cluster into Rancher and configures Longhorn and the PVC-key controller. The diagram shows the major dependencies, not every retry, wait, or API request.
A networking problem before Kubernetes even started
One early issue was the libvirt network itself. The script now handles migration from an older routed network (cluster-routed) to the NAT network (cluster-nat). It checks whether the network belongs to the lab, stops affected lab VMs before changing it, and refuses to remove a network used by an unrelated VM.
The script also checks access to QCOW2 images by the libvirt-qemu account. A VM disk can exist and still be unusable if QEMU cannot read the file or its backing image. These checks are small, but they matter for repeatable bootstrap.
I would rather have the script stop with a clear error than silently replace a disk or alter an unrelated VM.
RKE2, Cilium, and encrypted node-to-node traffic
For the downstream cluster I used RKE2 with Cilium instead of the default Canal configuration. The script disables the bundled ingress controller and kube-proxy, then supplies Cilium settings through RKE2's HelmChartConfig before the first server starts.
The relevant configuration is compact:
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: rke2-cilium
namespace: kube-system
spec:
valuesContent: |-
kubeProxyReplacement: true
k8sServiceHost: localhost
k8sServicePort: 6443
encryption:
enabled: true
type: wireguard
ingressController:
enabled: true
default: true
loadbalancerMode: shared
service:
type: NodePort
insecureNodePort: 30080
secureNodePort: 30443
A detail worth keeping: on an existing RKE2 installation, the script does not attempt to switch the CNI in place. It leaves the existing CNI and Helm configuration alone. Changing the CNI of a live cluster is a different migration problem.
In the tested deployment, all five nodes reached Ready. Cilium ran on every node, with two operator replicas. A Cilium encryption status check reported WireGuard, the cilium_wg0 interface, and four peers from one agent.
That confirms WireGuard was enabled and peers were present. It is not a claim that every possible inter-node or host-network traffic path was independently captured and verified.
Ingress and network policy
The Cilium Envoy Ingress configuration exposed HTTP and HTTPS through fixed NodePorts. The observed HTTP test returned 200 OK, with server: envoy, on port 30080 across all five node IPs.
A separate NetworkPolicy test allowed HTTP before an ingress-deny policy and timed out after it was applied. That is a useful functional check of policy enforcement, although it is not a comprehensive tenant-isolation audit.
Encrypted persistent storage with Longhorn
The next problem was storage. Multi-tenant applications need persistent volumes, and I wanted encryption enabled by default rather than relying on each application to request it.
The script installs Longhorn chart 1.13.0 using the RKE2 Helm controller. On the cluster nodes it prepares open-iscsi, cryptsetup, and other required packages. It then creates a default StorageClass named longhorn-encrypted, with three replicas and encryption enabled.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: longhorn-encrypted
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: driver.longhorn.io
allowVolumeExpansion: true
parameters:
numberOfReplicas: "3"
encrypted: "true"
csi.storage.k8s.io/node-stage-secret-name: ${pvc.name}
csi.storage.k8s.io/node-stage-secret-namespace: ${pvc.namespace}
csi.storage.k8s.io/node-publish-secret-name: ${pvc.name}
csi.storage.k8s.io/node-publish-secret-namespace: ${pvc.namespace}
csi.storage.k8s.io/node-expand-secret-name: ${pvc.name}
csi.storage.k8s.io/node-expand-secret-namespace: ${pvc.namespace}
Generating a key for each PVC
The key reconciler runs in the storage-security namespace and polls for PVCs using the encrypted StorageClass. It creates Secrets in the PVC namespace, not in its own namespace. This is a provisioning-time key mechanism; it does not implement off-cluster key backup or automatic key rotation.
A small Python controller watches for PVCs using longhorn-encrypted. For each one, it creates a Secret with the same name in the PVC's namespace. The key is generated using Python's secrets.token_urlsafe(48).
The controller also records the PVC UID in the Secret annotation storage.lab/pvc-uid. If the Secret already exists, it does not overwrite it. If the stored PVC UID differs from the current PVC UID, it logs a blocked mismatch rather than reusing the Secret.
That behavior is important. Replacing a volume's encryption key during an ordinary reconciliation would make its data inaccessible. Reusing a key after a PVC was deleted and recreated under the same name would also be an unsafe assumption.
The controller is intentionally simple: it polls PVCs every three seconds, has a single replica, and uses cluster-scoped permissions to list PVCs and read/create Secrets. Its list request is capped at 500 PVCs; if the API indicates pagination, the controller fails the scan rather than processing an incomplete list. It is a lab component, not a finished production key-management service.
Verifying the encryption, not just the YAML
The test went beyond checking the StorageClass flag:
- A 1 GiB PVC became
Bound. - Its encryption Secret was created automatically.
- A test Pod could write and read data.
dmsetupshowed acryptmapping.- The ext4 filesystem was mounted above the encrypted mapping.
Separate checks on multiple volumes also found different hashes for their generated key material. That supports the intended per-PVC key separation.
The distinction matters: declaring encrypted: "true" is configuration; observing the actual dm-crypt device and readable filesystem is evidence that the storage path works.
Re-running the bootstrap without destroying the cluster
For me, one of the most useful tests was running the bootstrap again against the existing environment.
The script distinguishes two modes:
| Mode | Intended behavior |
|---|---|
| Normal run | Reuse existing VMs and disks; reconcile the parts the script manages |
RECREATE=1 | Remove managed VMs and disks, then recreate them; destroys their data |
The normal rerun path preserves managed VM disks and the existing RKE2 CNI configuration. RECREATE=1 explicitly removes managed VM disks. A successful rerun therefore demonstrates preservation, not the ability to recover after destruction.
This is not just a convenience. A bootstrap that works once but destroys data on the second run is difficult to trust.
The normal rerun smoke test passed:
- All five RKE2 nodes remained
Ready. - Cilium agents remained
Running, without observed restarts. - The test PVC stayed
Bound. - The previously written test data remained readable.
Rancher also reported the imported cluster as Connected=True.
What I did not verify: a full destructive recreation followed by recovery of the original data. RECREATE=1 is an explicitly destructive path, not a disaster-recovery test. The verification record also does not establish every resource UID remained unchanged across the rerun.
What I learned
The most time-consuming work was not writing the Kubernetes resources. It was making the entire environment predictable:
Networking comes first. Libvirt NAT, host forwarding, SSH access, CNI selection, and ingress all interact. A working Kubernetes API does not prove applications are reachable.
Order matters. Cilium settings must be in place for the first RKE2 bootstrap. Changing a live CNI should not be treated as a routine rerun operation.
Storage encryption needs lifecycle rules. Creating a unique key is straightforward. Preserving it, refusing accidental replacement, and recovering it after cluster loss are the difficult parts.
Idempotency is observable behavior. A second successful script exit is not enough. I checked node readiness, Cilium state, PVC state, and whether existing data could still be read.
What v0.1.0 does not solve yet
The current implementation is a reproducible local foundation for a cloud-agnostic platform. It does not yet demonstrate deployments on multiple cloud providers.
Several areas remain open:
- Tenant provisioning: automatic namespaces, RBAC, policies, quotas, and application stacks.
- Tenant object storage: evaluating separate RustFS operator-managed instances and their bucket/credential isolation.
- Backup and recovery: Longhorn backups to an S3-compatible target, plus independent backup and recovery of encryption Secrets.
- Provider portability: Terraform/Ansible implementations and provider-specific networking/storage integration.
- Production hardening: pinned images and release checksums, secure credential handling, verified SSH host keys, external key management, and a production control-plane topology.
The existing RustFS service runs in a separate k3s cluster on the same physical host. It may be useful as an initial S3-compatible Longhorn backup target, but it is not an off-host disaster-recovery destination.
There is another important topology limitation: the tested RKE2 cluster has two server/etcd nodes. This is the observed lab configuration, not a recommended production HA quorum design.
Next steps
I froze Cluster Bootstrap v0.1.0 as the baseline for this experiment. The next step is to connect Longhorn backup to RustFS, save volume encryption keys independently, and prove that an encrypted volume can be restored with its original data.
After that, I want to move toward repeatable tenant provisioning and replace the host-specific VM provisioning with provider-aware infrastructure automation.
The point of this first version was not to build the whole platform in one script. It was to establish a working, testable base—and to find out which assumptions break when I deploy it again.
Technical basis: the frozen bootstrap.sh, README.md, and VERIFICATION.md from Cluster Bootstrap v0.1.0 (October 26, 2026). The verification results are observations from the running lab, not a fresh test performed for this article.