Upgrade the CosmicAC stack
Upgrade a running CosmicAC deployment to a new image release.
Upgrade a running CosmicAC deployment to a new image release.
In-cluster job images upgrade separately
task update upgrades only the Compose platform stack, not the GPU and inference images that run as jobs in your Kubernetes cluster. To change those, update their tags in the Kubernetes worker's config/common.json.
Prerequisites
You need the following before you start:
- A running CosmicAC deployment. See Deploy CosmicAC.
- The deployment repository on your host.
- The new image
TAGyou want to deploy.
Steps
Set the new tag
Set TAG=<newtag> in .env, or pass it inline in the next step.
Run the update
Pull the new images, add any new config keys, and recreate the stack at the new tag:
task updateTo skip editing .env, pass the tag inline with task update TAG=<newtag>. To scope the upgrade to one service, add SERVICES="cosmicac-app-node".
Change a config key's value (optional)
Do this step only if you want a different value for one of the new keys. task update sets each new key to the release default and leaves your existing values unchanged.
Find the key in the service's config/common.json, then set it in .env. This example sets pagination.limit to 100 for app-node:
APPNODE_COMMON_CONFIG__pagination__limit=100Apply the change, then restart the service:
task apply-app-node-common-config
task restart SERVICES="cosmicac-app-node"See Task deployment commands for each service's prefix and apply command, and for the other variable forms, including the ones that set values in config/facs/.
task update restarts the services it upgrades, so this restart applies only to your .env change. Any later change you make in .env needs both commands, because the apply command writes the config file without starting containers.
Verify the deployment
Confirm every service is up:
task psFor the full health checks, see Verify the deployment.
If a worker won't connect, refresh the shared runtime keys, then recreate the worker so it picks them up:
task wire
task restart SERVICES="cosmicac-wrk-server-k8s-nvidia"Roll back
To roll back, deploy the previous tag. The rollback preserves your state, but back up first for a snapshot:
task backup
task update TAG=<previousTag>