K8s Anti-Design Pattern Series – Combining clusters of production and non-production

Summary:

Even though Kubernetes was built from the ground up to orchestrate clusters, you shouldn’t make the mistake of using a monolithic cluster for everything. Two clusters are required at the very least: one for actual production and one for testing and development.  If you’re short on supplies, a bad idea would be to switch back and forth between production and non-production. However, developers report concerns about data security. All internal communication within a Kubernetes cluster is enabled by default unless specific measures are taken to disable it. Moreover, contrary to common belief, a pod in one namespace is able to freely communicate with a pod in another namespace. Some Kubernetes resources aren’t even namespaced.

To put it bluntly, Kubernetes namespaces are not a safety feature. Supporting multi-tenancy within a cluster and securing all workloads against each other is possible, provided your team has advanced Kubernetes knowledge. However, this is a time-consuming process, and in most cases, it is more convenient to set up a separate cluster for use during production.

There are many potential disasters that can arise from combining this anti-pattern with the one which talks about baking configuration inside containers.

• On a cluster that also holds production environments, a developer sets up a namespace for new features.
• They launch the namespace with their feature code and initiate integration tests.
• The integration tests either create fake data, purge the database, or otherwise interfere destructively with the backend.
• However, because of the containers’ internal production URLs and configuration, all integration tests actually impacted live workloads.
• In order to avoid this pitfall, it is simpler to divide clusters into production and non-production areas. Unfortunately, the official Kubernetes documentation has examples using prod/dev namespaces, and several tutorials imply that namespaces can be used for environment division.

The size of your business will determine how many clusters you have, but it’s important to remember that there could be more than two.

Production

• Lower-capacity copies of existing productions that look and feel the same
• Clusters of programmers working together to test new features 
• Cluster designed specifically for security and load testing 
• Grouping of in-house resources
• Maintaining a minimum of 2 (production and everything else that is not production).