Simplifying Kubernetes Deployments with Kustomize

In this article I will explain kustomize in detail. The content of the article;

• What is the kustomize?
• What are the kustomize features?
• Kustomize setup and configuration.
• Understanding Kustomize file structure,
• Kustomize Example

We want to deploy into Kubernetes cluster development, uat, staging, and production. We may have one or more environments. The configuration of each environment may have different deployment manifest. Each environment may have different cpu, ram and endpoints. Therefore, we need to configure each environment to have its own deployment manifest. Integration with kubectl is available for Kubernetes cluster management.

What is The Kustomize?

Kustomize is an open source configuration tool for Kubernetes. Provides deployment, statefulset, daemonsets, configmap and secret declarative definition and management of one or more environments without modifying the original yaml files. Kustomize has two basic concepts:

• Base
• Overlays

We can reuse it in all environments and add patches for each of these environments. Overlaying is the process of creating a customized version of the manifest file. In summary;

Base manifest + Overlay manifest = Customized Manifest Files

Note: Every environment needs to have a kustomization.yaml file.

What are the kustomize features?

The following are the key features of Kustomize;

• Kubernetes yaml has a declarative structure.
• The manifest file can change without changing the original file.
• Can add common tags and descriptions to all manifests.
• It can change container images and env depending on the env in which it is deployed.

Kustomize Install

Pre-requisites; Kubernetes cluster must be installed and running.

Kustomize module is integrated into kubectl but for kubernetes 1.14+.

Standalone Install: Kustiomize can be installed with the following command.

# Install
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash

# permission 
sudo install -o root -g root -m 0755 kustomize /usr/local/bin/kustomize

To check the installation;

kustomize version

Let’s Understand Kustomize

What are the kustomization concepts?

• Kustomization.yaml file: It is the main file used by the Kustomize tool. When you run the Kustomize build command, Kustomize looks for the file named kustomization.yaml. This file contains a list of all Kubernetes manifests (YAML files) that need to be managed by Kustomize. It also includes all custom customizations for the manifests that will be created. Kustomization manifest example;

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml
  - service.yaml

commonLabels:
  app: web

configMapGenerator:
- name: web-config
  files:
  - config/appsettings.json

secretGenerator:
- name: web-secret
  literals:
  - username=denizturkmen
  - password=denizturkmensecret

images:
- name: denizturkmen/python:latest
  newName: denizturkmen/python
  newTag: env1

• Base file: Represents the configuration that will be the same in all environments (dev, test and prod). All kubernetes manifests we use can add to base.
• Overlays file: It represents the configuration in which we make customizations on the manifests we use as default. With overlay we can make specific adjustments for each environment. This is where we overwrite or change all manifest values in our Default base folder.
• Patches: Patches or overlays are another method to manage our kubernetes configs. It provides more specific features for making changes to our manifests. In order to make changes, we need to use 3 parameters.

Operation Type: Add or remove or replace is used here.

Target: The resource name we want to change. 
  For example Deployment, Statefulsets etc

Value: The name of the value to be added or changed. 
  This is left blank for the removal operation.

There are 2 methods for patches.

JSON 6902 Patching: This method manage 2 things. These; target and patches details.

apiVersion: apps/v1
kind: Deployment
metadata:
  name:  web-deployment
  labels:
    name:  web-deployment
spec:
  template:
    metadata:
      labels:
        name:  web-deployment
  replicas: 1
    spec:
      containers:
      - image:  denizturkmen/python:latest
        name:  web-deployment
        env:
        - name:  deniz
          value:  turkmen       
        ports:
        - containerPort:  5000
          name:  web-deployment

The above deployment manifest example. The below patches manifest file example.

patches:
  - target:
      kind: Deployment
      name: web-deployment
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 3

When we apply patches, a replica of 3 instances is created.

Stragetic Merge Patching: The configuration config we provide in this path is similar to the standard kubernetes yaml. All we have to do is specify the areas we want to change. for example;

apiVersion: apps/v1
kind: Deployment
metadata:
  name:  web-deployment
  labels:
    name:  web-deployment
spec:
  template:
    metadata:
      labels:
        name:  web-deployment
  replicas: 1
    spec:
      containers:
      - image:  ipedrazas/docmock
        name:  web-deployment
        env:
        - name:  deniz
          value:  turkmen       
        ports:
        - containerPort:  5000
          name:  web-deployment

The above deployment manifest example. The below patches manifest file example.

patches:
  - patch: |-
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: web-deployment
      spec:
        replicas: 3

Patch From File: Instead of using both patch types via kustomization.yaml, we can make them separately via different files. To explain it a little better; In our kustomization.yaml file, we can give the patches file path where our patches configuration is located, write all your patches configurations to the relevant file and complete the process in the same way. for example; The following kustomization yaml file.

patches:
- path: replicas.yaml

The following replicas yaml file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-deployment
spec:
  replicas: 3

Transformers: As the name suggests, Transformers is a concept that adds or transforms the values we specify to a config. Using Transformers, we can update our basic Kubernetes yaml configurations to the values we specify. Some transformers and their uses are as follows;

• commonLabel: Adds a label to all Kubernetes manifests
• namePrefix: Adds a common prefix to all manifest files.
• nameSuffix: Adds a common suffix to all manifest files.
• Namespace: Adds a common namespace to all manifests.
• commonAnnotations: Adds a common annotations to all manifests.

Image transfer example;

apiVersion: apps/v1
kind: Deployment
metadata:
  name:  web-deployment
  labels:
    name:  web-deployment
spec:
  template:
    metadata:
      labels:
        name:  web-deployment
  replicas: 1
    spec:
      containers:
      - image:  denizturkmen/python:latest
        name:  web-deployment
        env:
        - name:  deniz
          value:  turkmen       
        ports:
        - containerPort:  5000
          name:  web-deployment

The below kustomization.yaml example;

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
  
images:
  - name: denizturkmen/python:latest
    newName: nginx
    newTag: latest

In the above example, the existing ipedrazas/docmock image ensures that the nginx image is not deployed with latest.

Deploying an Application Using Kustomize

Let’s make a kustomize example and let our scenario be as follows.

Specific python image needs to be deployed on dev and prod. We need a deployment with only 3 replicas in Dev, a Nodeport service, environment is dev env, and less ram and CPU resources. In prod, we need a deployment with 5 replicas, a LoadBalancer service, environment is prod env, different CPU and memory limits, a rolling update strategy and a service without Nodeport.

Directory structure;

├── kustomize
  ├── base
    │   ├── deployment.yaml
    │   ├── service.yaml
    │   ├── namespcace.yaml   
    │   ├── kustomization.yaml
    └ overlays
        ├── dev
        │   ├── deployment-dev.yaml
        |   ├── service-dev.yaml
        │   ├── namespcace.yaml    
        │   └── kustomization.yaml
        └── prod
            ├── deployment-prod.yaml
            ├── service-prod.yaml
            ├── namespcace-prod.yaml
            └── kustomization.yaml

Base file contain deployment, service and kustomization yaml. Here we keep the basic yaml that we use in common and from which we can override the environment. The below base namespace;

apiVersion: v1
kind: Namespace
metadata:
  name: base

The below base deployment.yaml;

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: python-flask
        image: nginx
        ports:
        - name: python-flask
          containerPort: 5000
---

The below base-service;

apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  selector:
    app: web
  ports:
  - name: http
    port: 5000

The below base-kustomization;

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
  
resources:
- deployment.yaml
- service.yaml
- namespace.yaml

Both base folder and kubernetes manifest file are finished. Now let’s look at environment-based overlays; There are a few rules in the overlays file that you did not patch. The first example will be with patch yaml file. Let’s patch our dev scenario so that the number of deployment replicas is 3 service type nodePort and dev namepsace.

Note: In overlays yaml, we only write the changes we want to change or add.

• Dev Overlays; The below namespace.yaml manifest;

apiVersion: v1
kind: Namespace
metadata:
  name: dev

• The below deployment.yaml manifest;

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-dev-deployment
spec:
  replicas: 3 # Update the replica count to 3
  template:
    spec:
      containers:
      - name: nginx
        image: denizturkmen/python:env1
        resources:
          limits:
            cpu: "200" # Lower CPU limit to 200m (0.2 CPU cores)
            memory: "256Mi" # Lower memory limit to 256 MiB
          requests:
            cpu: "100" # Lower CPU request to 100m (0.1 CPU cores)
            memory: "128Mi"
        env:
        - name: MY_VARIABLE
          value: "This is deployment environment....."

• The below service.yaml manifest;

apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  type: NodePort

Kustomization patch for dev environment. However, we will use the patch method from the file. kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

namePrefix: dev-
namespace: dev

patches:
- path: dev-service.yaml
- path: dev-deployment.yaml
# - path: dev-namespace.yaml

Note: Since namespace kustomization is not patched after kustomization version 5+, we must use namePrefix.

Additionally, JSON 6902 Patching using dev env is done as follows. But we will proceed with patched from file.

.
.
.


resources:
- ../base


patches:
  - target:
      kind: Deployment
      name: web-dev-deployment
    patch: |-
      - op: replace
        path: "/spec/containers/0/image"
        value: "denizturkmen/python:env1"
      - op: replace
        path: /spec/replicas
        value: 3 
      -  op: add
         path: "/spec/containers/0/env"
         value:
         - name: MY_VARIABLE
           value: This is deployment environment.....
      - op: replace
        path: "/spec/type"
        value: "NodePort"
      - op: replace
        path: "/spec/ports/0/nodePort"
        value: 30000

To check manifest command.

cd /kustomize/overlays/dev

kustomize build .

Note: After customization of version 5+, Base and overlays/dev manifest container name should have the same name for image update.

The following command is used to run kubernetes cluster directly.

kustomize build overlays/dev | kubectl apply -f -

or

kubectl apply -k overlays/dev

Output,

• Prod Overlays

Kustomization patches for environment. The following command will perform devloyment manifest example.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-deployment
spec:
  replicas: 5 # Update the replica count to 3
  template:
    spec:
      containers:
      - name: python-flask
        image: denizturkmen/python:env1
        resources:
          limits:
            cpu: "500m" # Lower CPU limit to 200m (0.2 CPU cores)
            memory: "512Mi" # Lower memory limit to 256 MiB
          requests:
            cpu: "300m" # Lower CPU request to 100m (0.1 CPU cores)
            memory: "256Mi"
        env:
        - name: MY_VARIABLE
          value: "This is production environment....."

Prod service manifest;

apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  type: LoadBalancer

Kustomization patch for prod environment.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

namePrefix: prod-
namespace: prod

patches:
- path: prod-service.yaml
- path: prod-deployment.yaml

Additionally, Stragetic Merge Patching prod env is done as follows. But we will proceed with patched from file.

Note: Another patches manifest. However, Manifest used under kustomize version 5.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

patches:
  - patch: |-
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: web-prod-deployment
      spec:
        replicas: 5 
        image: denizturkmen/python:ev1
        env:
        - name: MY_VARIABLE
          key: "This is Prod environment...."

Note: Another patches manifest. However, Manifest used under kustomize version 5.

apiVersion: kustomize.config.k8s.io/v1beta1

kind: Kustomization

# resources section contains the resources that kustomize is handling
resources:
  - deployment.yaml

namespace: prod

images:
  - name: denizturkmen/python:latest
    newName: denizturkmen/python
    newTag: "env1" 

To check manifest command.

kustomize build overlays/prod

Output; for Prod env;

The following command is used to run kubernetes cluster directly.

kustomize build overlays/prod | kubectl apply -f -

or

kubectl apply -k overlays/prod

Both dev and prod environment deployment are completed !!!!

Conclusion

The above section describes a real quick and easy to understand introduction of Kustomize and some sample code using Kustomize tool. The main advantage that I saw in Kustomize is that it is very easy to start using, if we compare this with other tools like Helm.

Referance:

Kubernetes native configuration management

GitHub - kubernetes-sigs/kustomize: Customization of kubernetes YAML configurations

Declarative Management of Kubernetes Objects Using Kustomize

Kubernetes Kustomize - JsonPatches6902 overview | Sergey Kryvets Blog