Upgrading

Standard upgrade

Check the Behavior changes index and the per-version section for any release between your current version and the target. These flag breaking changes, new values, and potential issues to address before running the recipe below.

Apply the updated CRDs before upgrading the Helm releases; Helm does not upgrade CRDs automatically:

helm show crds oci://ghcr.io/cosmonic/cosmonic-control --version 0.5.3 | kubectl apply -f -

Then upgrade the control plane with the same values file used during installation:

# Check the currently installed release and chart version
helm list -n cosmonic-system

# Upgrade to a new chart version
helm upgrade cosmonic-control oci://ghcr.io/cosmonic/cosmonic-control \
  --version 0.6.1 \
  --namespace cosmonic-system \
  -f cosmonic-control-values.yaml

Upgrade the HostGroup release separately:

helm upgrade hostgroup oci://ghcr.io/cosmonic/cosmonic-control-hostgroup \
  --version 0.6.1 \
  --namespace cosmonic-system

Wait for the rollout to complete before considering the upgrade done:

kubectl rollout status deploy -l app.kubernetes.io/instance=cosmonic-control -n cosmonic-system
kubectl rollout status deploy -l app.kubernetes.io/instance=hostgroup -n cosmonic-system

Chart version vs. appVersion

Chart versions do not track appVersion one-to-one. A single chart version (e.g. 0.4.1) may ship updated appVersion values as underlying Cosmonic Control images are patched. Check the current appVersion before upgrading:

helm show chart oci://ghcr.io/cosmonic/cosmonic-control --version 0.6.1

Rolling back

Rolling back the Helm release

To roll back to the previous release after a failed upgrade:

helm rollback cosmonic-control -n cosmonic-system
helm rollback hostgroup -n cosmonic-system

helm rollback restores the Helm release to its previous revision. Check the rollback status:

kubectl rollout status deploy -l app.kubernetes.io/instance=cosmonic-control -n cosmonic-system
kubectl rollout status deploy -l app.kubernetes.io/instance=hostgroup -n cosmonic-system

CRD rollback caveat

warning

Helm does not downgrade CRDs on rollback — by design, to prevent accidental data loss. If the failed upgrade included CRD changes, rolling back the Helm release will leave the newer CRD versions in place.

If you need to roll CRDs back to a previous version, apply the old CRD manifests manually:

# Pull CRDs from the target (older) chart version
helm show crds oci://ghcr.io/cosmonic/cosmonic-control --version <previous-version> | kubectl apply -f -

Verify the CRDs were restored before proceeding:

kubectl get crd | grep -E 'cosmonic|wasmcloud'

Behavior changes since v0.4.0

A scan of operator-visible changes by release. Each entry links to the upgrade section below where the change is explained in full. Releases not listed here ship only dependency bumps and internal fixes; see cosmonic/control releases for those.

Version	Change	Action
v0.6.0	HTTPTrigger and WorkloadDeployment expose /scale for HPA	None — opt in by creating a HorizontalPodAutoscaler
v0.6.0	HostGroup dataNats.url for control-plane / data-plane NATS split	Set if workload messaging should leave the control-plane NATS
v0.6.0	Nexus persistence switches to a StatefulSet; existingClaim removed	Migrate any pre-v0.6.0 nexus-data PVC onto data-nexus-0
v0.6.0	Nexus HA via nexus.replicas (odd, RAFT quorum)	Scale Nexus to 3/5 after upgrade for HA; the operator matches artifact-replicas automatically
v0.5.3	Minimum Kubernetes lowered to v1.24 (was v1.29)	None — relaxation only
v0.5.2	HTTPTrigger.spec.environment default is now unpinned (shared-host mode)	Set environment explicitly on triggers that should stay on a tenant-scoped hostgroup
v0.5.2	Nexus persistence opt-in (nexus.persistence)	Enable for durable wasi:keyvalue and Artifact JetStream state
v0.5.2	Operator NATS reconnect resilience	None — informational
v0.5.2	Workload RBAC scopes to operator.watchNamespaces	None unless using namespace-scoped install
v0.5.1	Metrics endpoint configurable; cluster-scoped metrics RBAC off by default	None unless you ran with secure metrics
v0.5.1	Host self-identification on telemetry	None — informational
v0.5.1	Rebuilt observability dashboards	Port any forked dashboards
v0.5.0	CA bundle support (opt-in)	None unless trusting a private CA
v0.5.0	Observability stack image bumps	Mirror new tags if air-gapped
v0.4.1	Default ingress provider switched to Traefik	Set ingress.provider: istio if you rely on Istio
v0.4.1	New request-timeout values	Remove any application-level timeout workarounds
v0.4.0	Traefik is now the default ingress	Set ingress.enabled: false if exposing Envoy directly
v0.4.0	Console and Cloud components removed	Remove console: / console_ui: blocks from values
v0.4.0	wasmCloud 2.0.3 and OpenTelemetry field changes	Verify Perses dashboards and OTel consumers

Upgrading from v0.3.x

Version 0.4.0 is a significant release. Review the changes below and update your values file before running helm upgrade.

Apply the updated CRDs

Helm does not upgrade existing CRDs automatically. Export the CRDs from the v0.4.0 chart and apply them manually:

helm show crds oci://ghcr.io/cosmonic/cosmonic-control --version 0.4.0 | kubectl apply -f -

The v0.4.0 CRDs add new optional fields (imagePullPolicy on hostgroup workloads, readOnly on volume mounts) and remain backward compatible with existing HTTPTrigger, WorkloadDeployment, and Host manifests.

Traefik is now the default ingress

The chart now deploys Traefik as the edge proxy by default and creates a Traefik IngressClass. The ingress Kubernetes Service defaults to ClusterIP and sits behind Traefik. See Ingress and Workloads for the full architecture.

If your v0.3.x install exposed Envoy directly via envoy.service.type: LoadBalancer or NodePort, preserve that pattern on v0.4.0 by setting ingress.enabled: false:

# cosmonic-control-values.yaml
ingress:
  enabled: false
envoy:
  service:
    type: LoadBalancer   # or NodePort, with httpNodePort set

Otherwise, switch to Traefik and migrate external routing to standard Kubernetes Ingress resources that target the ingress Service on port 80. The Traefik section of Ingress and Workloads has a worked example.

Console and Cloud components removed

The Helm chart no longer ships Console, Console UI, or the console: values block. When you run helm upgrade, Helm removes the existing Console Deployment, Service, and ConfigMap from the cluster. Perses is now the primary observability surface and continues to ship with the chart.

Remove any console: or console_ui: blocks from your values file before upgrading.

wasmCloud 2.0.3 and OpenTelemetry field changes

Cosmonic Control v0.4.0 upgrades the control-host image to wasmCloud 2.0.3, built on wash v2 internals. OpenTelemetry field names emitted by the host changed as part of that reconciliation. If you have Perses dashboards, alerting rules, or external OTel consumers that query on specific field names, verify them against the new host logs after the upgrade.

With the values file updated and CRDs applied, run the Standard upgrade flow with --version 0.4.0.

Upgrading from v0.4.0 to v0.4.1

Chart 0.4.1 ships appVersion 0.4.2. The chart was bumped once; the host image (ghcr.io/cosmonic/control-host) was patched twice, ending at 0.4.2. Upgrading installs the latest host image automatically.

Apply the updated CRDs

Helm does not upgrade existing CRDs automatically. Export the CRDs from the v0.4.1 chart and apply them manually:

helm show crds oci://ghcr.io/cosmonic/cosmonic-control --version 0.4.1 | kubectl apply -f -

The v0.4.1 CRDs add the following optional fields. Existing v0.4.0 manifests remain valid without changes.

• HTTPTrigger.spec.timeout (string, Go duration) — per-route upstream request timeout applied to the xDS route generated for the trigger. Bounds how long Envoy waits for a response from the backing Wasm host before returning upstream request timeout. Accepts values like 300s or 5m. When unset, Envoy's default of 15s applies.
• HTTPTrigger.spec.kubernetes.service.name and HTTPTrigger.spec.template.spec.kubernetes.service.name (string) — references an existing Kubernetes Service that the operator maintains an EndpointSlice for, pointing to the host pods running the workload. When set, the operator also registers DNS aliases (service-name, service-name.namespace.svc.cluster.local) with the host so cluster-internal callers can reach the workload via Service DNS without going through an external gateway.
• WorkloadDeployment.spec.template.spec.kubernetes.service.name (string) — same mechanism as above for non-HTTP workloads.
• HostInterface.name (string, lowercase alphanumeric and hyphens) on both CRDs — uniquely identifies an interface instance when multiple hostInterfaces entries share the same namespace and package. Components use this name as the identifier parameter in resource-opening functions (e.g. store::open(name)). Required when multiple entries of the same namespace:package exist.

Default ingress provider switched to Traefik

ingress.provider now defaults to "traefik". v0.4.0 defaulted to "istio". If your values file does not set ingress.provider explicitly and your environment expects Istio routing, set it before upgrading:

# cosmonic-control-values.yaml
ingress:
  provider: istio

If you migrated to Traefik during the v0.3.x → v0.4.0 upgrade or you run a fresh v0.4.0 install with Traefik, no action is required.

New request-timeout values

Two new chart values bound how long the host and the ingress will wait for a reply on long-running requests. The defaults are conservative (300s) and cover GPU compute jobs and other multi-minute request/reply patterns out of the box.

• nexus.requestTimeoutSeconds on cosmonic-control-hostgroup (default 300) — bounds any single wasmcloud:messaging/consumer.request call (and other NATS request/reply traffic). Set to null to fall back to the async-nats client default (10s).
• ingress.istio.workloadsRequestTimeoutSeconds on cosmonic-control (default 300) — per-route upstream request timeout for the workloads VirtualService when running with the Istio ingress provider. Set to null to fall back to Envoy's default (15s). For per-trigger overrides under either ingress provider, use the new HTTPTrigger.spec.timeout field.

If you previously worked around long-running requests by raising async-nats client timeouts at the application level or by editing VirtualService routes after install, you can remove those overrides.

With the values file updated and CRDs applied, run the Standard upgrade flow with --version 0.4.1.

Upgrading from v0.4.x to v0.5.0

Chart 0.5.0 ships appVersion 0.5.0 and bumps the control-host image to wasmCloud v2.2.1.

Apply the updated CRDs

helm show crds oci://ghcr.io/cosmonic/cosmonic-control --version 0.5.0 | kubectl apply -f -

CA bundle support (opt-in)

Both charts gain a top-level caBundle value group with mutually exclusive sources (contents, existingConfigMap, existingSecret) and per-component opt-in flags. Defaults are off, so existing installs render an identical manifest. See the Trusting a private CA tip on the Get Started page for the install-time pattern.

The freeform volumes / volumeMounts values on the hostgroup chart still work; the new mechanism is additive.

Observability stack image bumps

The observability sidecars are pinned to new tags in values.yaml. If you mirror images to a private registry, mirror the new tags before upgrading — see Air-Gapped Installation.

Component	v0.4.x tag	v0.5.x tag
prometheus	v3.3.1	v3.11.3
loki	3.5	3.7.2
tempo	2.8.2 (hardcoded)	2.10.5
otel-collector-contrib	0.120.0	0.152.0
envoy	v1.35.2	v1.38.0
jaeger	2.11.0	2.18.0
kiwigrid-k8s-sidecar	1.30.10	2.7.3

The tempo Deployment previously bypassed values.yaml and pulled a hardcoded 2.8.2. From v0.5.0 onward, tempo.image.tag drives the deployed tag like every other observability component.

With the values file updated and CRDs applied, run the Standard upgrade flow with --version 0.5.0.

Upgrading from v0.5.0 to v0.5.1

Chart 0.5.1 ships appVersion 0.5.1.

Apply the updated CRDs

helm show crds oci://ghcr.io/cosmonic/cosmonic-control --version 0.5.1 | kubectl apply -f -

Metrics endpoint is now configurable; cluster-scoped metrics RBAC is off by default

The cosmonic-control chart gains two new values on a new metrics block:

• metrics.secure (default false) — when true, the operator serves metrics over HTTPS with delegated authn/authz, and the chart installs the cluster-scoped metrics-reader / metrics-auth ClusterRoles and binding required for that mode. When false (the default), metrics are served as plain HTTP with no auth and no metrics ClusterRole is created.
• metrics.port (default 8081) — the operator's metrics bind port. Drives the container port, Service port, and the in-cluster OpenTelemetry collector's scrape target in lockstep.

Previous chart versions installed those ClusterRoles unconditionally, but the operator was never started with secure metrics, so they were unused. helm upgrade removes them. The metrics endpoint behavior is functionally identical to before, minus the unused RBAC.

Host self-identification on telemetry

The hostgroup now exports k8s.pod.name, k8s.pod.uid, k8s.node.name, k8s.namespace.name, and cosmonic.io/hostgroup as OpenTelemetry resource attributes via the Kubernetes downward API. The collector drops the k8sattributes processor and its ClusterRole — the host self-identifies, so the collector no longer needs cluster-scope pod read.

Loki indexes k8s_pod_name, k8s_namespace_name, and cosmonic_io_hostgroup as structured-metadata fields on log records. Use these in LogQL queries to scope logs to a specific pod, namespace, or HostGroup.

Rebuilt observability dashboards

The HTTPTrigger Detail dashboard is rebuilt and renamed to Workload Activity (namespace/workload/host variables, per-host RPS and error rate, separate HTTP/Blobstore/Keyvalue/Messaging/Logs rows, Tempo Explorer drill-in links). A new Host Activity dashboard surfaces per-host span rates and a host-scoped logs panel keyed off k8s_pod_name.

If you forked the previous dashboard, port your customizations against the new dashboard definitions. See Observability.

With the values file updated and CRDs applied, run the Standard upgrade flow with --version 0.5.1.

Upgrading from v0.5.1 to v0.5.2

Chart 0.5.2 ships appVersion 0.5.2 and bumps the control-host image to wasmCloud v2.3.0.

Apply the updated CRDs

helm show crds oci://ghcr.io/cosmonic/cosmonic-control --version 0.5.2 | kubectl apply -f -

HTTPTrigger.spec.environment now controls scheduler scope

In prior versions, an HTTPTrigger with spec.environment unset would only schedule workloads onto same-namespace hosts. As of v0.5.2, an unset environment leaves the workload unpinned — the scheduler treats every host as fair game, including any shared hostgroup. This change matches the field's documented intent and unblocks the shared-host use case, but it is a behavior change for any cluster that relied on the previous implicit namespace pinning.

If you have triggers that should remain confined to a tenant-scoped hostgroup, set spec.environment explicitly on those triggers before upgrading. See Pinning to a host environment for the field reference.

Nexus persistence is configurable

Nexus stores its NATS JetStream state under /data. By default the chart still mounts /data as an emptyDir, so JetStream state — including every wasi:keyvalue bucket created by workloads — is lost when the Nexus pod restarts or reschedules.

A new nexus.persistence block on the cosmonic-control chart enables a PersistentVolumeClaim for /data:

# cosmonic-control-values.yaml
nexus:
  persistence:
    enabled: true
    size: 10Gi               # PVC size
    storageClassName: ""     # empty = cluster default StorageClass
    existingClaim: ""        # set to reuse a pre-provisioned PVC

Enable this if any of your workloads rely on wasi:keyvalue for durable state, or if you use the Artifact CRD's JetStream Object Store backing for offline image caches.

Operator NATS reconnection is now resilient

The operator's NATS client previously gave up after 60 reconnect attempts. It now retries reconnection indefinitely and exits cleanly on a permanent disconnect, letting Kubernetes restart the pod. No configuration change is required, but expect to see nats disconnected / nats reconnected log lines during NATS or Nexus pod recycles where previously the operator stayed silent.

Workload RBAC can be scoped to watched namespaces

When operator.watchNamespaces is set, the operator's runtime.wasmcloud.dev RBAC is now installed as a Role + RoleBinding per watched namespace instead of a single cluster-wide ClusterRole. Operators watching all namespaces (the default) see no change. See Namespaced runtime.wasmcloud.dev RBAC.

With the values file updated and CRDs applied, run the Standard upgrade flow with --version 0.5.2.

Upgrading from v0.5.3 to v0.6.0

Chart 0.6.0 ships appVersion 0.6.0 and bumps the control-host image to wasmCloud v2.4.0.

Apply the updated CRDs

helm show crds oci://ghcr.io/cosmonic/cosmonic-control --version 0.6.0 | kubectl apply -f -

The v0.6.0 CRDs add the /scale subresource on HTTPTrigger and WorkloadDeployment and an admission-time regex on allowedHosts. Existing manifests remain valid; spec.replicas now defaults to 1 if omitted.

Horizontal Pod Autoscaling for HTTPTrigger

Both HTTPTrigger and WorkloadDeployment now expose the Kubernetes /scale subresource, so a stock HorizontalPodAutoscaler can target either resource directly. The trigger's status.currentReplicas and status.selector mirror the backing WorkloadDeployment, so HPA reads the live replica count without traversing another resource. See Autoscaling for a worked manifest and a community-call demo. This is additive — setting spec.replicas manually still works the same way.

Separate control-plane and data-plane NATS connections

The HostGroup chart gains a dataNats.url value. When set, the wasmCloud host opens a second NATS connection used by wasi:keyvalue, wasi:blobstore, and wasmcloud:messaging, while the existing nexus.url connection continues to carry scheduling and host lifecycle traffic. Leaving dataNats.url empty (the default) routes workload traffic over the control-plane connection as before, so existing releases are unaffected.

# hostgroup-values.yaml
dataNats:
  url: "nats://workload-nats.shared.svc.cluster.local:4222"

See Separating the data-plane NATS for when this helps and what to watch for.

Nexus persistence is now a StatefulSet, with optional HA

nexus.persistence.enabled: true now switches Nexus from a single-replica Deployment to a StatefulSet with a per-pod volumeClaimTemplate. The default behavior with persistence.enabled: false is unchanged — a single-pod Deployment backed by an emptyDir.

Three things move together when persistence is on:

• A new nexus.replicas value controls the StatefulSet size. The chart fails the install at template time if the count is even or if replicas > 1 is combined with persistence disabled, so misconfiguration cannot reach the cluster. Use an odd count (1, 3, 5) so JetStream's RAFT meta-group can form a quorum.
• The operator passes -artifact-replicas={{ nexus.replicas }} so the ARTIFACTS JetStream object store is replicated to match. The operator reconciles the bucket's replica count on every start, so scaling Nexus from 1 to 3 brings the Artifact bucket along.
• The nexus.persistence.existingClaim value is removed. The StatefulSet manages its PVCs via volumeClaimTemplate (data-nexus-0, data-nexus-1, ...); a pre-provisioned nexus-data PVC from v0.5.2 or v0.5.3 is not migrated automatically.

If you turned on persistence in v0.5.2 or v0.5.3, the v0.6.0 upgrade replaces the standalone Deployment + nexus-data PVC with a StatefulSet + data-nexus-0 PVC. JetStream state on the old PVC does not roll forward on its own. To preserve it, copy /data from the previous PVC into data-nexus-0 (for example via a one-shot Job that mounts both) before scaling the StatefulSet up, or accept that wasi:keyvalue buckets and the Artifact bucket will be recreated empty. If your values file sets existingClaim, remove it before upgrading.

For high availability, scale nexus.replicas to 3 (or 5) after the upgrade has settled. JetStream KV buckets and Artifact stores must additionally be created with replicas >= 3 to actually survive a node failure — clustering the servers alone does not replicate R=1 data.

Other HostGroup additions

The HostGroup chart adds three pass-through values, each empty by default:

• env — appended after the chart's own environment variables, so entries can reference earlier ones via $(VAR) and override them on duplicate names. Standard Kubernetes env shape, including valueFrom.
• envFrom — populates the host container from one or more ConfigMaps / Secrets.
• extraArgs — appended verbatim to the host CLI, for host flags the chart does not template.

These are additive; existing releases see no change.

With the values file updated and CRDs applied, run the Standard upgrade flow with --version 0.6.0.

Upgrading from v0.6.0 to v0.6.1

Chart 0.6.1 ships appVersion 0.6.1 and pins the upstream wasmCloud release to v2.4+, which switches CRD reconciliation from a client-side get-create-update to a server-side upsert. The change eliminates the "object already exists" reconciliation race that previously surfaced under contention. No user action — the behavior fix lands automatically with the chart bump.

Apply the updated CRDs

helm show crds oci://ghcr.io/cosmonic/cosmonic-control --version 0.6.1 | kubectl apply -f -

The v0.6.1 CRD schemas are unchanged from v0.6.0; the apply is a no-op for the API but kept in the upgrade flow for consistency.

Operator log level is now configurable

Previous releases hard-coded the operator's log level to debug, which produced noisy logs in steady state. The cosmonic-control chart adds a new operator.logLevel value with a default of info:

# cosmonic-control-values.yaml
operator:
  logLevel: info  # one of: debug, info, warn, error

After upgrade, the operator emits at info by default. Set operator.logLevel: debug in your values file to restore the previous behavior — useful when triaging reconciliation issues.

Traefik CRDs install only when Traefik is the configured ingress provider

In prior releases, the Traefik CRDs shipped unconditionally. The v0.6.1 chart wraps that block in {{- if and .Values.ingress.enabled (eq .Values.ingress.provider "traefik") }}, so Traefik CRDs are installed only when ingress.enabled: true and ingress.provider: "traefik". Installs with ingress disabled, or with a non-Traefik provider configured, no longer carry the unused CRDs after upgrade.

Most clusters won't notice — the Cosmonic Control chart defaults to Traefik when ingress is on. If you had ingress disabled but were relying on the Traefik CRDs being present for other workloads, install them out of band (e.g., from the Traefik chart) before upgrading.

Runtime operator leader election is unconditional

The runtime operator now always runs with leader election enabled, regardless of replica count. The behavior matches what production deployments already did via Helm values; it's now baked in. No configuration change.

With the values file updated and CRDs applied, run the Standard upgrade flow with --version 0.6.1.