Unable to change flavor in CloudscaleMachineTemplate for a running cluster

[nickspizzas]

Description
Changing a topology variable that patches a CloudscaleMachineTemplate field (e.g. cloudscaleWorkerMachineFlavor) breaks CAPI reconciliation on ClusterClass-based clusters.

Error
error reconciling the Cluster topology: failed to reconcile MachineDeployment dev/dev-01-md-0-xxxxx: failed to create patch helper for CloudscaleMachineTemplate dev/dev-01-md-0-xxxxx: server side apply dry-run failed for modified object: admission webhook "vcloudscalemachinetemplate-v1beta2.kb.io" denied the request: CloudscaleMachineTemplate.infrastructure.cluster.x-k8s.io "dev-01-md-0-xxxxx" is invalid: spec.template.spec: Forbidden: field is immutable

Environment

• CAPCS version: v1.0.1
• CAPI version: v1.13.4
• Kubernetes version: v1.35.6

Steps to reproduce

1. Create a cluster.
2. After the cluster is fully provisioned, change the variable value in Cluster.spec.topology.variables (e.g. change flavor from flex-8-4 to flex-16-8) and apply.
3. Observe the CAPI controller log — it immediately enters a retry loop with the error above.
   No new CloudscaleMachineTemplate is created; no MachineDeployment rollout is triggered.

Expected behavior
CAPI creates a new CloudscaleMachineTemplate, updates the MachineDeployment reference, and triggers a rolling node replacement automatically.

Actual behavior
Cluster is not being reconciled and no new nodes are provisioned.

Notes
You mentioned some limitations here: https://github.com/cloudscale-ch/cluster-api-provider-cloudscale/blob/main/docs/troubleshooting.md#webhook-validation-common-rejections

Should this also apply to CloudscaleMachineTemplate? And if yes, how can I change the Flavor of a running Cluster?

[mweibel]

Thanks for the report.

After a quick review, it looks like we didn't notice dry-run support in the contract when implementing cluster class support. I'll be verifying this in more detail later but it looks like we'll need to implement a condition similar to what CAPI DockerMachineTemplate does:
https://github.com/kubernetes-sigs/cluster-api/blob/9e9f71ea379daf37989cd7b4fffbc8cb497481cb/test/infrastructure/docker/internal/webhooks/devmachinetemplate.go#L76-L86

[nickspizzas]

Thank you for looking into it.

dry-run support in the contract

Yes, this looks like the key - Allowing dry-run SSA

FYI - I used following workaround:

# export template
kubectl get cloudscalemachinetemplates.infrastructure.cluster.x-k8s.io dev-01-md-0-xxxxx -oyaml

# remove template
kubectl delete cloudscalemachinetemplates.infrastructure.cluster.x-k8s.io dev-01-md-0-xxxxx

# re-apply template with adjusted flavor and --server-side --field-manager=capi-topology 
kubectl cat << _EOF_ | kubectl apply --server-side --field-manager=capi-topology -f -
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: CloudscaleMachineTemplate
metadata:
  name: dev-01-md-0-xxxxx
  namespace: dev
  labels:
    cluster.x-k8s.io/cluster-name: dev-01
    topology.cluster.x-k8s.io/deployment-name: md-0
    topology.cluster.x-k8s.io/owned: ""
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta2
    kind: Cluster
    name: dev-01
    uid: <cluster-uid>
spec:
  template:
    spec:
      flavor: flex-8-2 # adjust with new flavor
      image: custom:xxx-kube-v1.35.6
      rootVolumeSize: 20
      serverGroup:
        name: dev-01-md-0
_EOF_

# delete worker nodes 1 by 1 and it will be re-created with new flavor
kubectl delete machine <worker-node>

[mweibel]

FYI v1.1.0 is released with this fix.