Overview

We will create a Talos Linux Kubernetes cluster running on Proxmox using Terraform infrastructure as code (IAC).

Prerequisites

• Proxmox Virtualization
• Terraform CLI Installed
• Helm CLI Installed
• Kubectl CLI Installed
• Talosctl CLI Installed

Create a Terraform Directory

mkdir k8s-tf-example
cd k8s-tf-example
touch main.tf

Add Proxmox and Talos providers

Edit main.tf, change the endpoint to match your Proxmox URL.

terraform {
  required_version = ">= 1.3"

  required_providers {
    proxmox = {
      source = "bpg/proxmox"
      version = "~> 0.111.1"
    }
    talos = {
      source = "siderolabs/talos"
      version = "~> 0.11.0"
    }
  }
}

provider "proxmox" {
  endpoint = "https://192.168.1.21:8006/"
  insecure = true
}

Set Proxmox Secret Environment Variables

Put in your root password for Proxmox. If you'd like you can create another user as long as they have sufficient permissions. If so use that username as well.

export PROXMOX_VE_USERNAME="root@pam"
export PROXMOX_VE_PASSWORD="super-secret"

Create Talos Cluster Module

This uses an open source Terraform module developed by BB Tech Systems - talos. If you encounter issues or have feature requests please create an issue on the Github repo.

Append module and output blocks to main.tf. Change the module inputs to match your needs. In particular change the control_nodes and worker_nodes maps to match your desired VM names(keys) and proxmox node names(values).

module "talos" {
    source  = "bbtechsys/talos/proxmox"
    version = "1.2.0"
    talos_cluster_name = "test-cluster"
    talos_version = "1.13.9"
    control_nodes = {
        "test-control-0" = "pve1"
        "test-control-1" = "pve1"
        "test-control-2" = "pve1"
    }
    worker_nodes = {
        "test-worker-0" = "pve1"
        "test-worker-1" = "pve1"
        "test-worker-2" = "pve1"
    }
}

output "talos_config" {
    description = "Talos configuration file"
    value       = module.talos.talos_config
    sensitive   = true
}

output "kubeconfig" {
    description = "Kubeconfig file"
    value       = module.talos.kubeconfig
    sensitive   = true
}

Spreading a cluster across Proxmox nodes

The example above puts every VM on pve1. Since v1.0.0 the module downloads the Talos image onto each Proxmox host that needs one, so a cluster can span your whole Proxmox cluster just by naming different hosts:

    control_nodes = {
        "test-control-0" = "pve1"
        "test-control-1" = "pve2"
        "test-control-2" = "pve3"
    }

There is nothing to configure for this. The module reads the ISO datastore's shared flag from Proxmox: a shared datastore (NFS, CIFS, CephFS) is downloaded to once and used by every host, while a node-local one such as the default local gets its own copy per host.

Other options worth knowing

Three additions in v1.1.0, all optional and all defaulting to the behavior above:

• A highly available API endpoint. By default the cluster endpoint is the first control node's IP, which is a single point of failure. talos_cluster_endpoint points it at a Talos VIP or a load balancer instead.
• Static addresses with no DHCP. Set talos_platform = "nocloud" and declare control_node_addresses / worker_node_addresses. Proxmox builds a cloud-init drive and Talos takes its address from it at boot, so nothing needs to hand the node an address first. Nodes also get their Terraform name as their Kubernetes node name rather than a generated one.
• Per-node config patches. control_machine_config_patches_by_node and its worker equivalent apply patches to a single node, appended after the shared patches.
• Waiting for the cluster to be ready. wait_for_cluster_health = true makes apply block until the cluster is serving, so the kubeconfig works the moment Terraform returns. See the apply step below.

The module documentation covers each of these with examples.

Already using an older version of the module?

v1.0.0 is a breaking change. The Talos image resource was renamed and is now keyed by Proxmox host, which makes each VM's boot disk id unknown at plan time — and that forces replacement, so upgrading in place destroys and recreates every VM unless you add a moved block first. The 1.0 upgrade guide walks through it. Going from 1.0 to 1.1 needs no action, and 1.1 to 1.2 only if you had a malformed talos_cluster_endpoint or a per-node patch key matching no node — both of which 1.2 now catches at plan time instead of silently ignoring. The changelog covers those, and every other release.

Always read terraform plan before applying an upgrade. If you see your VMs listed under must be replaced, stop.

Terraform Init

terraform init

Terraform Apply

terraform apply

Grab a cup of coffee and give Talos a few minutes to bootstrap. You can monitor the progress in the Proxmox console for a control node. When it says STAGE Running, KUBELET Ready you can proceed.

If you would rather not watch the console — or you are running this in CI, where terraform apply && kubectl apply -f ... would otherwise race the cluster coming up — set wait_for_cluster_health = true on the module and Terraform will not return until the cluster is actually serving:

    wait_for_cluster_health = true

It is off by default because the check runs on every terraform plan, not just when creating the cluster, so a node being down for maintenance would block planning. Turn it on for the first build and for CI. If you disable the built-in CNI to install something like Cilium, add cluster_health_skip_kubernetes_checks = true as well — otherwise the nodes stay NotReady until you deploy that CNI and the wait can never finish.

Download the Client Configurations and Test

terraform output -raw kubeconfig > kubeconfig
terraform output -raw talos_config > talos_config.yaml
export KUBECONFIG=$(pwd)/kubeconfig
export TALOSCONFIG=$(pwd)/talos_config.yaml

If you wish to make this permanent run:

mkdir ~/.talos
cp talos_config.yaml ~/.talos/config
mkdir ~/.kube
cp kubeconfig ~/.kube/config

Test your config.

kubectl get nodes

You should see something like:

NAME            STATUS   ROLES           AGE     VERSION
talos-4vt-cmm   Ready    control-plane   5m21s   v1.36.0
talos-dxq-l3o   Ready    <none>          5m24s   v1.36.0
talos-exw-zz9   Ready    <none>          5m15s   v1.36.0
talos-g1d-91g   Ready    <none>          5m21s   v1.36.0
talos-s1z-dgw   Ready    control-plane   5m14s   v1.36.0
talos-zmm-jgf   Ready    control-plane   5m18s   v1.36.0

talosctl containers

You should see something like:

NODE            NAMESPACE   ID                     IMAGE   PID    STATUS
192.168.1.210   system      apid                           2122   RUNNING
192.168.1.210   system      ext-qemu-guest-agent           1938   RUNNING
192.168.1.211   system      apid                           2088   RUNNING
192.168.1.211   system      ext-qemu-guest-agent           1934   RUNNING
192.168.1.209   system      apid                           2094   RUNNING
192.168.1.209   system      ext-qemu-guest-agent           1939   RUNNING
192.168.1.213   system      apid                           2097   RUNNING
192.168.1.213   system      ext-qemu-guest-agent           1936   RUNNING
192.168.1.213   system      trustd                         2140   RUNNING
192.168.1.208   system      apid                           2098   RUNNING
192.168.1.208   system      ext-qemu-guest-agent           1940   RUNNING
192.168.1.208   system      trustd                         2146   RUNNING
192.168.1.212   system      apid                           2096   RUNNING
192.168.1.212   system      ext-qemu-guest-agent           1935   RUNNING
192.168.1.212   system      trustd                         2144   RUNNING

Install KubeRay Operator (optional, will be used in future blog posts)

For future blog posts, we will be exploring running Ray for Machine Learning, Distributed SQL using DataFusion Ray, and Reinforcement Learning using Ray RLLib. This will install kuberay, a Kubernetes operator that will make it possible to run Ray in Kubernetes.

From raycluster quick start:

helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm repo update

# Install both CRDs and KubeRay operator v1.2.2.
helm install kuberay-operator kuberay/kuberay-operator --version 1.2.2

# Confirm that the operator is running in the namespace `default`.
kubectl get pods
# NAME                                READY   STATUS    RESTARTS   AGE
# kuberay-operator-7fbdbf8c89-pt8bk   1/1     Running   0          27s

Destroy (if desired)

WARNING - This will delete everything we just created. Use this only if you want to start over.

terraform destroy