K8s Anti-Design Pattern Series – Manual kubectl edit/patch deployments

Summary

The issue of configuration dispersion predates Kubernetes and has been around for a long time. It occurs when two or more identical environments no longer share the same setup due to ad hoc deployments or adjustments.This issue becomes more pressing over time, to the point where it may become necessary to reverse-engineer a running instance to determine the machine’s original setup.

Using Kubectl

This issue can arise in Kubernetes as well. The ‘kubectl’ tool has a lot of flexibility because it has in-built apply, edit, and patch commands that can modify resources on a running cluster.Unfortunately, developers can readily abuse this technique. Changes made on the fly in the cluster are never documented outside of it. Failure to properly configure the setting is a common cause of unsuccessful deployments. If the configuration of the production environment has changed from that of the testing environment, the deployment will fail in production even though it succeeded in staging.

Adjustments during deployment

It’s simple to fall into this mistake. As a rule, ad hoc adjustments are made for the sake of making hotfixes, “quick solutions,” or other dubious hacks.It is strongly recommended that manual distributions not be performed using Kubectl. The GitOps paradigm suggests that all deployments be managed by the deployment tool and, ideally, documented there.

Git – Deployments

If you exclusively use Git commits for deployments:

• Git change history provides a comprehensive record of cluster activity.
• At any given moment, you are fully aware of the precise contents of each cluster, as well as the ways in which their respective environments vary from one another.
• Reading the Git configuration is a simple way to clone or recreate an entire system.
• Simply resetting your cluster to an earlier commit will restore its prior configuration.
• Most significantly, in the event of a failed deployment, you can quickly identify the most recent change that affected it and how that change impacted its configuration.

All editing and patching done with kubectl should be considered experimental. Attempting to make manual resource changes to a cluster in production is extremely risky. It’s important to have a well-thought-out deployment process, but you should also concur with your colleagues that this misuse of kubectl is unacceptable.