Mastering Kustomize: Advanced Techniques for Kubernetes Configuration Management

Python3 Example — Abstraction

Introduction

I will explain how we can do advanced and abstraction together with kustomize. I will explain what Kustomize is, why Kustomize is used, Kustomize file structure diagram and advanced Kustomize example.

What is the Kustomize

Kustomize is an open-source tool designed to customize Kubernetes application configurations. Unlike traditional templating tools, Kustomize allows you to manage Kubernetes objects through a concept called "overlays," enabling you to apply modifications to YAML manifests without altering the original files. This approach promotes reusability and maintainability of configuration files.

Why Use Kustomize?

1. Declarative Management: Kustomize allows you to manage your Kubernetes objects in a declarative manner. This means you define the desired state of your system using YAML files, and Kustomize helps you achieve that state without needing to modify the original files.
2. Environment-Specific Configurations: In most real-world scenarios, you need different configurations for different environments (development, staging, production). Kustomize enables you to maintain a single base configuration and apply environment-specific changes using overlays. This approach reduces duplication and simplifies maintenance.
3. No Templating Language: Unlike tools like Helm that use templating languages, Kustomize works directly with YAML files. This eliminates the need to learn a new syntax and keeps the Kubernetes manifest files simple and easy to understand
4. Patch Management: Kustomize provides powerful patching mechanisms, allowing you to make targeted changes to specific resources without altering the entire configuration. This is useful when you need to override or customize certain fields, such as image tags or resource limits, for different environments.
5. Composability: Kustomize encourages reusability by allowing you to define “bases” (common configurations) and “overlays” (environment-specific modifications). You can compose complex configurations by stacking overlays on top of each other.
6. Idempotence: Kustomize ensures that applying the same configuration multiple times produces the same result, which is critical for maintaining consistent deployments across environments.

Kustomize Standalone Install: Kustomize can be installed with the following command.

# Install
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash

# permission 
sudo install -o root -g root -m 0755 kustomize /usr/local/bin/kustomize

To check the installation;

kustomize version

Kustomize File Structure for Python Project

├── kustomize
  ├── base
    │   ├── deployment.yaml
    │   ├── service.yaml
    │   ├── namespace.yaml
    │   ├── kustomization.yaml
    └ overlays
        ├── dev
        │   └── kustomization.yaml
        │   └── configmap-dev-env
        │   └── secret-dev-env.env
        └── uat
        │   └── kustomization.yaml
        │   └── configmap-uat-env    
        │   └── secret-uat-env.env

Before moving on to our example, I would like to explain a few important points. Let’s think about it, we wrote a micro service and we deploy this micro service with CI. There are two or more environments. The point that distinguishes environments from each other is environment variable and config volume.

Manage your infrastructure more efficiently and effectively with customized base manifests. Below, you’ll find the file structure and manifests for the base manifests.

You can create the following folder structure using the mkdir -p command:

mkdir -p /path/to/your/project/

mkdir -p kustomize

First, I’ll create the namespace.yaml file to define the namespace for our resources.

apiVersion: v1
kind: Namespace
metadata:
  name: base

Next, I’ll create the service.yaml file to establish the service for our resources.

apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  selector:
    app: web
  ports:
  - name: http
    port: 5000

Next, I’ll create the deployment.yaml file to establish the deployment for our resources.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: python
  namespace: default
  labels:
    app: web
spec:
  selector:
    matchLabels:
      app: web
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: web
    spec:
      serviceAccountName: default
      containers:
      - name: python
        image: denizturkmen/python:X.XX.X
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
          limits:
            cpu: 100m
            memory: 100Mi
        livenessProbe:
          tcpSocket:
            port: 5000
          initialDelaySeconds: 5
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 3
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /
            port: 5000
          initialDelaySeconds: 5
          timeoutSeconds: 2
          successThreshold: 1
          failureThreshold: 3
          periodSeconds: 10
        envFrom:
          - secretRef:
              name: env-secret
        ports:
        - containerPort: 5000
          name: python
        volumeMounts:
        - name: python-volume
          mountPath: /app/config  
      volumes:
        - name: python-volume
          configMap:
            name: python-env
      restartPolicy: Always

I will explain the deployment manifest above in detail.

1. envFrom: Since we have multiple environments, use envFrom to load the entire environment from a file rather than specifying each name-value pair separately.
2. volumeMounts: We use volumeMounts to load custom microservice configurations.

Next, I’ll create the kustomization.yaml file to establish the kustomize for our resources.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: kustomize-advance

resources:
- namespace.yaml
- service.yaml
- deployment.yaml

Our base manifests for Kustomize are ready. Now, let’s create the environment-specific custom files.

Let’s create the Kustomize file for the dev environment, along with the secret generator and volume config generator to read the environment variables.

First, I’ll create the kustomization.yaml file to define the kustomize for our resources.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

namePrefix: dev-
nameSuffix: "-001"
namespace: dev

secretGenerator:
  - name: env-secret
    envs:
      - secret-dev-env.env

configMapGenerator:
  - name: python-env
    envs:
      - configmap-dev-env

images:
- name: denizturkmen/python
  newTag: "env1"

As shown above, environment-based abstraction is achieved effectively. Typically, patching methods like JSON 6902 and patches merge are used under overlays for environments like DEV, UAT, and STAGING. However, these methods limit abstraction and are not much different from Helm templates. Instead, we should consolidate all processes into a single Kustomize file. In this approach, we read environment variables from secretGenerator, configure volumes from configGenerator, and manage image changes through new tags.

Next, I’ll create the secret-dev-env.env file to establish the secretGenerator for our resources.

MY_VARIABLE="This is DEV environment....."

Next, I’ll create the configmap-dev-env file to establish the configMapGenerator for our resources. configMapGenerator was used to create a fully representative example.

username="deniz"
surname="turkmen"

Let’s build with Kustomize. Navigate to the dev environment folder and run the build command.

kustomize build .
or
kubectl kustomize .

Use the following command to deploy for dev-environment.

kubectl apply -k .

Now, let’s connect to the pod to verify the secretGenerator environment variables and configMap. For this;

kubectl exec -it pod_name -n namespace -- bash

Let’s also check the output when we send a service request.

curl http://cluster_ip:port_number

As you can see, we received the correct output.

We’ll create only the kustomization.yaml, configmapGeneratorand secretGenerator for the UAT environment. First, let's review the secretGenerator content and name it secret-uat-env.env.

MY_VARIABLE="This is UAT environment....."

Next, I’ll create the configmap-uat-env file to define the configMap for our resources.

username="deniz"
surname="turkmen"
envrionment="UAT"

Next, I’ll create the kustomization.yaml file to define the kustomize for our resources.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

namePrefix: uat-
nameSuffix: "-001"
namespace: uat

secretGenerator:
  - name: env-secret
    envs:
      - secret-uat-env.env

configMapGenerator:
  - name: python-env
    envs:
      - configmap-uat-env

images:
- name: denizturkmen/python
  newTag: "env1"

Let’s build with Kustomize. Navigate to the uat environment folder and run the build command.

kustomize build .
or
kubectl kustomize .

Use the following command to deploy for uat-environment.

kubectl apply -k .

Now, let’s connect to the pod to verify the secretGenerator environment variables. For this;

kubectl exec -it pod_name -n namespace -- bash

Let’s also check the output when we send a service request.

curl http://cluster_ip:port_number

As you can see, we received the correct output.

Conclusion

In this guide, we’ve systematically walked through the process of creating and managing environment-specific configurations using Kustomize. We’ve emphasized the importance of environment-based abstraction, showcasing how to effectively handle configurations without relying on complex patching methods. Instead, by consolidating all operations into a single Kustomize file, we’ve streamlined the deployment process, ensuring that environment variables, configuration volumes, and image tags are handled efficiently.

We started by creating the necessary kustomization.yaml and secretGenerator files, specifically for the UAT environment, ensuring that our setup is both flexible and scalable. By using the secretGenerator, we were able to securely manage environment-specific secrets, and the structured approach we've taken allows for easy expansion as new environments are added.

Overall, this approach not only simplifies the deployment process but also enhances the maintainability and clarity of our configurations. Whether you’re deploying to development, UAT, or production, the principles we’ve covered will help ensure a smooth and consistent workflow.

Referance:

K8S-Full-Content-Notes/Kustomization/0- PythonApp at main · denizturkmen/K8S-Full-Content-Notes

Deniz Türkmen

Deniz Türkmen — Turkey | Professional Profile | LinkedIn

denizturkmen — Overview

DeNiZ Turkmen - Blog