Planning k8s on hetzner for my webapps

 

I have always been a fan of rke2 but one thing I found lacking and not yet added to their tool is not adding a floating IP for master HA cluster. I use to do the following https://github.com/MrAmbiG/k8s_rke2_HA_KubeVip in such cases in my k8s clusters but getting floating an IP for this and then another for my metalLB, pay for them separately, manage them, seemed like something which I should avoid. So, I decided to go the non non agnostic way of doing these 2 things in my setup where I will be going against my lift and shift principle in IaC and go with hetzner specific config for my HA master config and ingress controller LB setup.

Infrastructure Blueprint

Architecture: Hybrid Cloud (VM masters + Bare Metal workers).

• Operating System: openSUSE MicroOS (Immutable hosts).
• HA & Networking: Hetzner Cloud Load Balancers for API and Ingress; Hetzner vSwitch for private networking.
• Storage: Rook-Ceph for high-speed local block storage on workers.
• DevOps Management: Dedicated management/jump VM within Hetzner Cloud.

Phase 1: Hetzner Account & Server Prep

1. Generate API Token: Create a project in the Hetzner Cloud Console and generate a Read/Write API token.
2. SSH Key: Add your public SSH key to the "Security" tab.
3. Order Hardware:
   1. Order 3 small Cloud VMs (e.g., CPX11) for the Control Plane (masters).
   2. Order 1 small Cloud VM (e.g., CX11) for the Jump Server (DevOps).
   3. Order 3+ Bare Metal Servers (AX/EX line) for worker nodes. Ensure they have at least two identical raw drives each.

4. Setup vSwitch: Create a Private Network (vSwitch) in the console and attach all 7+ servers (VMs and Bare Metals) to it.
5. Security: Restrict public SSH/API access to only the public IP of your new jump server VM.

Phase 2: Deployment via Kube-Hetzner & Terraform

1. Configure Jump VM: Log in to your new jump server VM. Install terraform, kubectl, helm, and git.
2. Clone Project: Clone the Kube-Hetzner Terraform module.
3. Configure terraform.tfvars:
   1.  Set os_type = "microos".
   2.   Define control_plane_count = 3 using the VM instance types.
   3. Provide the IDs of your bare metal servers as worker nodes.
   4. Ensure configuration uses the vSwitch and enables the CCM and CSI.
   5. Configure the module to provision the Hetzner Cloud Load Balancer for the K8s API endpoint (this will be the IP in your kubeconfig).

4. Initialize & Deploy: Run terraform apply. This handles OS installation, networking, and cluster bootstrapping.
5. Verify Taints: Confirm masters have node-role.kubernetes.io/control-plane:NoSchedule taint, and bare metal workers are untainted.

Phase 3: Storage Layer (Rook-Ceph)

1. Keep Drives Raw: Ensure the second drive on all bare metal workers is unformatted.
2. Install Rook Operator: Deploy the official Helm chart from your jump server.
3. Provision OSDs: Create a CephCluster CRD to utilize the second raw drive on the worker nodes only.
4. Define Storage Classes: Use the created rook-ceph-block (RWO) and rook-cephfs (RWX) for your applications.

Phase 4: Ingress & Load Balancing

1. Install NGINX Ingress: Deploy NGINX Ingress Controller via Helm.
2. Configure Managed LB: Use the standard K8s Service of type: LoadBalancer with the necessary Hetzner annotations (e.g., load-balancer.hetzner.cloud/type: "lb11") to automatically provision a second, public Hetzner Cloud Load Balancer instance.
3. External Access: Configure your DNS records to point to the public IP address of this new managed load balancer.

Phase 5: WebApps Deployment

1. Containerize Apps: Build and push application Docker images to a registry.
2. Deployments & Services: Create K8s Deployment or StatefulSet manifests (which land on untainted workers).
3. Define Ingress Rules: Use standard K8s Ingress resources, utilizing the NGINX Ingress Controller to route traffic from your public hostname to internal services.
4. Persistent Data: Use PVCs with your rook-ceph StorageClasses to attach high-speed persistent storage for databases/uploads.