The Azure deploy that lied about being healthy
The first version of this incident treated one Azure state as the truth. That was the bug in the article as well as the deployment.
Azure Container Apps exposes several useful but different facts: ARM provisioning, revision running state, replica count, readiness and liveness probes, traffic weight, DNS reachability, and the application's own health endpoint. None proves all the others.
What “deployed” can mean
An ARM deployment can succeed after Azure accepts the desired resource shape. That does not prove a new container started, became ready, received traffic, or served a real request.
A revision can have a replica while its readiness state is not useful. A shared application hostname can serve traffic while a guessed per-revision hostname does not resolve in the selected revision mode. A healthy endpoint can return 401 because the probe used an authenticated route.
The status was not lying. We were asking it to answer a larger question than it represented.
Probes are application contracts
Container Apps supports startup, liveness, and readiness probes. Startup answers whether initialization completed. Liveness answers whether the process should be restarted. Readiness answers whether the replica should receive traffic.
Those endpoints should be anonymous to the platform but reveal no sensitive data. Readiness may include dependencies required to serve requests; liveness should not restart a healthy process because a remote dependency briefly failed.
With ingress, Azure can add default TCP probes, but an explicit HTTP probe is the only way to assert application-specific readiness. Microsoft documents the current defaults and revision behaviour in Health probes in Azure Container Apps.
The deploy gate we use now
A useful pipeline proves the states in order:
- The infrastructure deployment completed.
- The expected immutable revision exists.
- The revision provisioned and reports no platform error.
- Its readiness probe succeeds.
- Traffic points at the intended revision.
- The shared application hostname serves the expected application response.
- Logs remain clean through startup and the first real request.
In single-revision mode, test the shared application FQDN. Do not construct a revision hostname and assume DNS will publish it.
Secrets are another false green
Updating a Container App secret changes application configuration, but a running container resolves secret-backed environment variables at startup. A secret-only update may leave the active replica using the old value until a new revision or explicit restart.
So “the secret resource changed” and “the application consumed the new secret” are two separate checks.
Observe the first minutes
A deploy is not finished at the first successful request. Tail startup logs through the first scale event and one representative application call. Watch for crash loops, probe failures, dependency timeouts, authentication mistakes, and configuration that is read only after startup.
Keep rollback mechanical. Revisions are immutable snapshots, so record the previously healthy revision and the traffic state before changing either. In multiple-revision mode, move traffic only after readiness succeeds. In single-revision mode, know whether rollback means reactivating an older revision or deploying its image and configuration again.
The observation window should be condition-based, not a fixed sleep. Wait for the expected revision, readiness, traffic, and application response, each with its own timeout and diagnostic output. A five-minute pause that ends green tells less than a thirty-second sequence of explicit checks.
The lesson
Cloud control planes report state at their boundary. Reliable deployment requires a chain of evidence crossing ARM, the scheduler, probes, routing, DNS, and the application.
Name each assertion precisely. A green provisioning state is valuable. It is simply not a user request.