You are investigating an API deployment, and the repository does not match the cluster:
Git says:
replicas: 3
image: api:v2.4.1
Cluster says:
replicas: 5
image: api:v2.4.0
Which one is correct?
In GitOps, Git normally represents the desired state: what the team intends to deploy. The Kubernetes API exposes the live objects, including their current configuration and reported status. When they disagree, the difference can reveal a delayed rollout, an emergency change or a controller doing its job.
GitOps Kubernetes monitoring starts by seeing both sides and asking why they differ. Before setting replicas back to three or forcing a sync, establish which revision and environment you are comparing, who manages the resource, and whether the difference is intentional.
Desired state and live state answer different questions
Think of the deployment path like this:
Git repository
|
| desired state, after rendering if needed
v
GitOps controller
|
| applies configuration
v
Kubernetes API
|
| live objects and reported status
v
Cluster workloads
The GitOps controller compares intended configuration with live objects and reconciles differences according to its policy. Kubernetes workload controllers then work to make the running system match those objects.
The simple comparison is:
Desired ───┐
↓
Compare → Diff
↑
Live ──────┘
A Deployment's live spec.replicas: 5 means its current target is five replicas. It does not prove five Pods are Ready. Likewise, its Pod template's image does not prove every running Pod has already switched to that image during a rollout.
Read status, ReplicaSets and Pods when the question is what is actually serving requests. Desired/live comparison explains configuration; workload observations explain how far execution has progressed.
Read a small, meaningful difference first
Here are excerpts from the same Deployment, with identity fields omitted for clarity. They are examples to compare, not complete manifests to apply.
Desired:
spec:
replicas: 3
template:
spec:
containers:
- name: api
image: example/api:v2.4.1
Live:
spec:
replicas: 5
template:
spec:
containers:
- name: api
image: example/api:v2.4.0
With minus representing desired and plus representing live, the important differences are:
- replicas: 3
+ replicas: 5
- image: example/api:v2.4.1
+ image: example/api:v2.4.0
These are two investigation threads, not necessarily one mistake. Check whether an autoscaler or incident response changed replicas. Separately, check whether the new image revision was detected, rendered and applied successfully.
Capture the commit and time of the comparison. If the branch moves while you investigate, “Git says” can quietly change meaning halfway through the conversation.
Why drift happens
Kubernetes configuration drift describes a disagreement with intended configuration, but a raw difference is only evidence to interpret. Start with plausible causes before deciding which side to change.
Manual kubectl changes
Someone may have run:
kubectl scale deployment api --replicas=5
That changes the live replica target without editing Git. Treat the command as an example of how drift arises, not a recommended first response. In a real operation, specify the intended context and namespace explicitly.
If Git still declares three replicas, the settings disagree. Whether Argo CD changes the value back depends on its sync configuration and field ownership arrangements; the presence of GitOps alone does not mean immediate reversion.
Emergency production changes
During an incident, increasing capacity or rolling back an image may be intentional. A later engineer who sees only the diff can mistake the mitigation for a defect and undo it.
Check the incident record and responsible team. Decide whether to commit the temporary setting, restore the original intent or let another controller own the field. Record an expiry or follow-up so a justified exception does not become unexplained permanent drift.
Sync is delayed or disabled
The controller may not have detected the commit yet, automated sync may be disabled, or a sync window may prevent application. An Application can point to an older pinned revision even while the repository's default branch has moved on.
Argo CD's automated sync and self-heal settings are related but distinct. Self-heal controls automatic correction of live changes; do not infer it is enabled merely because automated deployment is configured. Check the actual Application policy in the automated sync documentation.
Reconciliation failed
The controller may know the desired state but be unable to apply it. A missing CRD, rejected admission request, insufficient permissions or a rendering error can stop progress.
Inspect the Application's conditions and operation result before retrying. A new sync will not fix a manifest that the API consistently rejects. Also distinguish an apply failure from a successful apply followed by an unhealthy rollout: they require different investigations.
Controllers and Kubernetes modify objects
Not every difference means a human made a mistake. Kubernetes defaults omitted fields, admission webhooks can inject settings, and controllers continuously update status.
An HPA can change a Deployment's replica target in response to load. Before restoring a Git value, check whether autoscaling is expected and whether your GitOps configuration deliberately leaves replicas under HPA control. Two controllers trying to own the same setting can repeatedly undo each other's work.
Injected sidecars and operator-managed fields need similar care. Understand who writes a field before deciding that its absence from a source file makes it unwanted.
Image and tag differences
An older image in the live Pod template may indicate an unapplied revision or an intentional rollback. Check the owning Application's target revision and the Deployment rollout before assuming a manual change.
Matching tags are not proof of matching image contents if tags can be moved. When investigating the exact binary running, inspect Pod container status and image IDs or digests as well as the declared image string. Immutable image references make that question easier to answer.
Environment-specific overlays
A base file might declare three replicas while the production Kustomize overlay sets five. A Helm chart may obtain its image from a values file or an Application parameter override.
Compare the rendered configuration for the correct environment and revision. Reading the first file named deployment.yaml is not enough when another step changes the resource before it reaches Kubernetes.
Git is intended truth, not a complete incident report
The useful distinction is:
Git = intended configuration
Kubernetes API = live configuration + reported status
GitOps normally treats the repository as the intended source of truth. That makes reviewed changes reproducible and gives the team a history of intent. It does not make every commit operationally correct or explain every live deviation.
If Git accidentally reduces capacity, the right correction may be a reviewed Git change. If an unapproved live edit changed an image, the right correction may be reconciliation. If autoscaling owns replicas, the right correction may be to clarify field ownership rather than force either value.
“Make the diff disappear” is therefore an incomplete objective. The aim is to restore an understood, safe relationship between intent, ownership and running behaviour.
Argo CD: health and sync are different questions
An Argo CD Application describes a set of resources and where their desired configuration comes from. Its sync and health states help separate configuration alignment from resource condition.
Synced means live state matches the target under the configured comparison rules. OutOfSync means the comparison finds a difference. Healthy means the relevant resource health checks are satisfied; Degraded means those checks identify a problem. These are useful states, not the complete list of possible statuses. See Argo CD's core concepts.
Two combinations are particularly useful during incidents:
Synced + Degraded
Configuration matches; the workload has a health problem.
OutOfSync + Healthy
Configuration differs; current resource health checks pass.
For the first, a Deployment can match Git while its rollout has exceeded its progress deadline. Repeatedly syncing the same configuration is unlikely to fix an image that cannot start.
For the second, an older deployment can keep serving while a newer intended image has not been applied. Health alone does not prove the expected release is running. Conversely, a green resource health assessment does not replace request-level monitoring or SLO checks.
Check the revision and freshness of the comparison. A refresh asks Argo CD to compare again; a sync changes resources toward the target. Read the result of the former before deciding whether the latter is appropriate.
Why a raw YAML diff is noisy
A live object contains fields that a source manifest normally should not specify:
metadata.resourceVersion
metadata.uid
metadata.managedFields
metadata.creationTimestamp
status
resourceVersion changes as the API object changes. uid identifies this particular object instance. Creation time and field-management records describe its history, while status records controller observations. Comparing all of them against an intentionally small manifest can bury a one-line image change in pages of unrelated output.
A meaningful desired-vs-live comparison removes known bookkeeping noise from the configuration view and accounts for understood defaults. It should still preserve access to status for investigation; excluding status from a configuration diff does not make status unimportant.
Normalisation also requires restraint. Ignoring every annotation, injected field or replica difference can hide something worth investigating. Use targeted rules whose purpose the team understands, and inspect unknown differences rather than assuming they are harmless.
Argo CD supports scoped diff customisation, including selected paths and field managers. Those rules affect what “matching” means, so review them when a clean sync status seems inconsistent with a visible change. The diff customisation guide explains the available controls.
A practical investigation workflow
Follow the evidence from the live resource back to intent:
Resource looks wrong
↓
Check live Kubernetes object
↓
Check GitOps ownership
↓
Find source manifest
↓
Render desired state if necessary
↓
Compare Desired vs Live
↓
Determine reason for drift
↓
Fix Git or reconcile safely
Establish the target and current behaviour
Confirm customer, context and namespace first. For example, these read-only checks inspect a named production target:
kubectl --context=customer-a-prod -n payments get deployment api -o yaml
kubectl --context=customer-a-prod -n payments describe deployment api
kubectl --context=customer-a-prod -n payments get hpa
Then inspect the relevant Pods, events and logs to understand the impact. A live resource watch helps follow changes during the investigation, but a current view does not retain all historical evidence.
Find ownership and the exact source
Look for Argo CD tracking information and the owning Application. Do not confuse a Kubernetes ownerReference, such as a Pod's ReplicaSet owner, with the GitOps Application that manages the deployment.
Read the Application's destination, source repository, path, target revision and sync/health information. In a central Argo CD installation, its Application may live in a different management cluster. Missing permissions or an absent local Application is not proof that the workload is unmanaged.
Render before comparing when necessary
Use the same source revision, tool versions, values and parameters as the deployment process. For a Kustomize environment, an inspection command might be:
kubectl kustomize overlays/production > desired.yaml
That renders local files; it does not apply them. For Helm, obtain the rendered manifests or reproduce the chart version, release name, namespace, values files and parameter overrides. A local approximation with different inputs is not authoritative desired state.
Compare matching resource identities and retain the original output for reference. Keep sensitive manifest data within the customer's approved workflow.
Explain the difference before writing
Check the sync result, recent commits, autoscalers and incident changes. Field management records may identify a writer, but they are not a complete audit trail; use available audit records when attribution matters.
Choose a reviewed Git correction or an approved reconciliation based on that explanation. Check the selected revision and whether the operation includes pruning, which can delete resources. Afterward, verify both configuration alignment and workload health rather than treating a successful sync request as completion.
Viewing Git and live state together
Biebie Kube helps investigate GitOps state next to the Kubernetes resource. It can resolve Argo CD ownership from available cluster information, show Application sync and health, and identify repository source information.
On request, its source comparison locates a supported manifest and compares it with the live object. The implementation removes common API bookkeeping fields and handles a defined set of defaults and controller-related differences. That makes the comparison easier to read without claiming to understand every possible Kubernetes mutation.
There is an important boundary: a source file is not always rendered desired state. Biebie Kube's source lookup reports cases that require rendering or cannot be matched unambiguously; it does not universally render Helm, Kustomize or arbitrary deployment pipelines. Repository access and visibility of the owning Application also affect what it can establish.
Argo CD remains responsible for reconciliation. Biebie Kube brings resource and GitOps context together so an engineer can understand the difference before deciding what to change.
For this investigation workflow in a desktop Kubernetes client, explore Biebie Kube.