Skip to content

How to Manage Kubernetes Clusters for Multiple Customers

BiebieKubernetes, Operations

Most Kubernetes advice assumes your clusters belong to you. You have a dev cluster, a staging cluster and a production cluster, they share a naming scheme, and the worst thing that can happen is you run a command against staging when you meant dev.

Consultants, managed service providers and contractors work under a different set of constraints. The clusters belong to different organisations, the naming schemes were decided by people who have never met each other, credentials arrive by whatever channel that customer prefers, and a mistake is not an internal embarrassment but a conversation with someone else's incident process.

This article covers the patterns that hold up under those constraints.

Do not merge every kubeconfig into one file

The instinct when you collect kubeconfigs is to merge them:

KUBECONFIG=~/.kube/config:~/Downloads/customer-a.yaml \
  kubectl config view --flatten > ~/.kube/merged
mv ~/.kube/merged ~/.kube/config

This works, and it is a trap.

A merged file has one namespace for context names, so two customers who both named their cluster production collide, and the loser gets silently renamed or overwritten depending on how you merged. It has one current-context, so the notion of "where am I" is global across every customer you support. And it grows without bound: when an engagement ends, removing that customer's entries from a 2,000-line merged file is error-prone enough that most people simply do not, which means expired credentials for organisations you no longer work with sit in the same file as live ones.

Keep one file per customer instead:

~/.kube/
  config                 # your own clusters, if any
  customers/
    acme.yaml
    globex.yaml
    initech.yaml

KUBECONFIG accepts a list, so you can still merge at the point of use when you actually need cross-cluster visibility:

export KUBECONFIG=~/.kube/config:$(find ~/.kube/customers -name '*.yaml' | paste -sd:)

The important part is that the merge is a runtime view, not a stored artefact. Deleting a customer is deleting a file.

Make context names carry the information you need

The default context name from a cloud provider tells you almost nothing useful in a multi-customer setting. EKS gives you the cluster ARN. GKE gives you gke_project_region_cluster. Neither says whose it is or how much care it deserves.

Rename contexts to a scheme you control, with the customer first and the environment explicit:

kubectl config rename-context \
  arn:aws:eks:ap-southeast-1:123456789012:cluster/prod-main \
  acme-prod

A scheme like customer-environment sorts sensibly, tab-completes usefully, and puts the two facts you most need — whose cluster, and how dangerous — in the first thing you read. If you use a prompt that shows the current context, this is also the string that will be sitting in front of you when you type a delete command.

Set a default namespace per context

A context can carry a namespace, and setting it removes a whole class of mistake where a command lands in default because you forgot -n:

kubectl config set-context acme-prod --namespace=acme-platform

This matters more across customers than within one organisation, because the namespace conventions differ. One customer puts everything in namespaces named after teams, another mirrors their Git repository structure, a third uses default for genuinely everything. Encoding each customer's convention into their context means you stop carrying it in your head.

Separate credentials from cluster definitions

Kubeconfig files can embed credentials directly, and for customer clusters they frequently do — a service account token pasted into the users section, or a client certificate encoded inline. That means the file itself is a secret, which makes it awkward to back up, sync between machines or share with a colleague taking over the engagement.

Where the customer's setup allows it, use an exec credential plugin instead, so the file describes how to obtain a credential rather than containing one:

users:
  - name: acme-prod
    user:
      exec:
        apiVersion: client.authentication.k8s.io/v1
        command: aws
        args:
          - eks
          - get-token
          - --cluster-name
          - prod-main
          - --region
          - ap-southeast-1

The kubeconfig becomes a description of how to reach the cluster, and the actual secret stays wherever your cloud CLI or identity provider keeps it. That also means the token is short-lived rather than a long-lived string sitting in a file.

Treat the network path as part of the cluster definition

Customer clusters are usually not reachable from the open internet. Getting to them means a VPN, and each customer has chosen a different one. In practice, "connect to Acme's cluster" is a two-part operation: bring up Acme's VPN with the right identity, then use Acme's kubeconfig.

The failure mode is subtle. If the VPN is down, your kubectl command does not fail with "you are not connected to the right network" — it fails with a connection timeout, which looks identical to a cluster that is genuinely unreachable. Time spent debugging an API server that is fine is time wasted.

Two things help. First, record which VPN profile and which login identity belong to which cluster, in the same place you record the cluster, so the association is written down rather than remembered. That is the job Biebie Access exists to do. Second, prefer tooling that distinguishes a network-level failure from a TLS failure from an authentication failure, so the error tells you which layer to look at.

Mark production and make it behave differently

Every customer has clusters you can experiment with and clusters you cannot. The difference is invisible in kubectl — the same command runs the same way regardless of what is on the other end.

At minimum, make production visible in your shell. A prompt segment that shows the current context, coloured by environment, is a small change with a large effect, because the information arrives without you asking for it.

Better still, make destructive actions on production require something you cannot do by muscle memory. Biebie Kube lets you mark a cluster as production and then requires the resource name to be typed out in full before a destructive action runs. Typing payments-api deliberately is a different cognitive act from clicking a button you have clicked a hundred times on staging.

Prefer optimistic concurrency when editing live objects

Editing a resource directly in a customer cluster is sometimes unavoidable — an incident, a misconfiguration to correct, a value to check. The risk is not that you make a mistake in the YAML. It is that someone else, or a controller, changed the object between when you read it and when you wrote it, and your write discards their change without either of you noticing.

Kubernetes has a mechanism for this. Every object carries a resourceVersion, and an update that includes the version you read will be rejected with a conflict if the object has moved on:

kubectl get deployment payments-api -o yaml > payments.yaml
# edit payments.yaml, leaving metadata.resourceVersion intact
kubectl replace -f payments.yaml

If the object changed in between, you get Operation cannot be fulfilled ... the object has been modified rather than a silent overwrite. Note that kubectl apply does not behave this way by default — it merges, and a merge can absorb someone else's change without a conflict. Any tool that offers a YAML editor should be sending the resourceVersion you loaded; if it does not, it is capable of losing changes you never saw.

When the engagement ends, archive rather than delete

Finished engagements produce a specific kind of clutter: credentials that no longer work, contexts you will never select again, and the small ongoing risk of selecting one by accident.

Deleting them is the obvious move, and it is usually wrong. Six months later someone asks what version of an operator that customer was running, or the engagement restarts, and the configuration you deleted has to be requested again from people who may have moved on.

Move it out of the active set instead. A separate directory that is not in your KUBECONFIG path achieves this with plain files. The point is that inactive access should require a deliberate act to become active again, without the information being destroyed.

What this adds up to

None of this is exotic. One file per customer, context names that say whose cluster and how dangerous, namespaces encoded in the context, credentials obtained rather than stored, the network path recorded next to the cluster, production visibly marked, edits that fail loudly on conflict, and finished work archived rather than deleted.

The reason to be deliberate about it is that multi-customer work removes the safety net that single-organisation work has. There is no shared convention to fall back on, no colleague who would have caught it, and no internal blast radius. The structure has to come from you.

Biebie Kube was built around these patterns — clusters grouped by customer, kubeconfigs read where they already are, production clusters marked and guarded, and edits that carry their resource version. It is free for macOS and Windows.