Enhance Your Modern CI/CD Workflow with Gitlab, Kustomize, Kubernetes and Argo-CD Integration

Python Example

I will explain two different examples of GitOps architecture:

1. Deploying from the same branch in both the project repository and the GitOps repository. Although this structure may conflict with a kustomized setup, I find it useful for branch-to-branch deployments. Project development branch — > gitops development branch.
2. In our second architecture, the goal is to push a new image to the environment defined in the kustomization overlays file across all environments whenever a build is triggered from this project. The GitOps repository will have a single branch, which is the master branch. The idea here is to manage the newly created image in a single, streamlined process, which is preferred by developers for its simplicity.

We have two architectures: one designed by me and the other by my dear friend Bahadır Geçgel. I won’t say which is whose, but we are very curious about your feedback. Which architecture do you think is the most accurate, useful, and logical? Your comments are truly important to us, so we would appreciate it if you could provide your thoughts with detailed explanations.

Introduction

Recently, I encountered and successfully resolved a challenge related to better separating and optimizing CI/CD processes. Although Jenkins was used on the CI side, I preferred Gitlab for this approach. Therefore, I used Gitlab for the CI process and wrote Kubernetes manifests using Kustomize. On the CD side, I performed deployments with ArgoCD. This approach allows us to manage the CI/CD pipeline in a more organized, efficient, optimized, and clear manner, resulting in a streamlined deployment process.

Pre-requisites;

• Kubernetes needs to be installed on the operating system.
• The ArgoCD CLI needs to be installed on the GitLab runner.
• ArgoCD needs to be installed on Kubernetes.
• Installing Kustomize-CLI is optional.
• Jq or Yq needs to be installed on the Gitlab runner.

The most important point we need to know is: git branch logic, we will build gitops on it.

If you don’t have Kubernetes set up in your environment, you can follow the instructions in this article to install it.

ArgoCD-CLI Install: ArgoCD-CLI can be installed with the following command.

# Install
curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64

# permission
sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd

#Removing
rm argocd-linux-amd64

Kustomize Standalone Install: Kustomize can be installed with the following command.

# Install
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"  | bash

# permission 
sudo install -o root -g root -m 0755 kustomize /usr/local/bin/kustomize

# check
kustomize version

If you don’t have ARGOCD set up in your environment, you can follow the instructions in this article to install it.

If you don’t have jq and yq set up in your environment, you can follow the instructions in this article to install it. For this, jq

sudo apt update

sudo apt install -y jq

For this, yq

sudo apt update

sudo apt install -y yq

I designed an end-to-end strategy like this:

• The development branch’s development kustomization manifesto should be updated.
• The UAT branch’s uat kustomization manifesto should be updated.
• The Staging branch’s staging kustomization manifesto should be updated.
• The Production branch’s production kustomization manifesto should be updated.

Secrets related to the project should not be stored in the Git repository. Instead, they should be stored in secret management systems such as HashiCorp Vault or cloud providers like AWS, Azure, and GCP.

Firstly, I will create two projects on Gitlab: The first repository will be for a Python example, and the second repository will be for GitOps.

Secondly, I will create gitops projects on Gitlab: We will create development, uat, staging, and production branches for the GitOps project.

Kustomize File Structure for GitOps Repository

├── kustomize
  ├── base
    │   ├── deployment.yaml
    │   ├── service.yaml
    │   ├── namespace.yaml
    │   ├── kustomization.yaml
    └ overlays
        ├── development
        │   └── kustomization.yaml
        │   └── secret-dev-env.env
        └── uat
        │   └── kustomization.yaml   
        │   └── secret-uat-env.env
        └── staging
        │   └── kustomization.yaml 
        │   └── secret-staging-env.env
        └── production
        │   └── kustomization.yaml
        │   └── secret-production-env.env
        │   ── ── ── ── ── ── ── ── ── ── ── 

Let’s start by writing a kustomization manifest for the development environment and then push it to the GitOps repository. But first, let’s create our manifests in a base directory that will be shared across all environments. This base will be used in every environment, but in the article, I will only demonstrate the example for the development environment.

First, I’ll create the namespace.yaml file to define the namespace for our resources.

apiVersion: v1
kind: Namespace
metadata:
  name: base

Next, I’ll create the service.yaml file to establish the service for our resources.

apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  selector:
    app: web
  ports:
  - name: http
    port: 5000

Next, I’ll create the deployment.yaml file to establish the deployment for our resources.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: python
  namespace: default
  labels:
    app: web
spec:
  selector:
    matchLabels:
      app: web
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: web
    spec:
      serviceAccountName: default
      containers:
      - name: python
        image: denizturkmen/python:X.XX.X
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
          limits:
            cpu: 100m
            memory: 100Mi
        livenessProbe:
          tcpSocket:
            port: 5000
          initialDelaySeconds: 5
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 3
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /
            port: 5000
          initialDelaySeconds: 5
          timeoutSeconds: 2
          successThreshold: 1
          failureThreshold: 3
          periodSeconds: 10
        envFrom:
          - secretRef:
              name: env-secret
        ports:
        - containerPort: 5000
          name: python
      restartPolicy: Always

I will explain the deployment manifest above in detail.

1. envFrom: Since we have multiple environments, use envFrom to load the entire environment from a file rather than specifying each name-value pair separately.

Next, I’ll create the kustomization.yaml file to establish the kustomize for our resources.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

metadata:
  name: devops-deniz

resources:
- service.yaml
- namespace.yaml
- deployment.yaml

Our base manifests for Kustomize are ready. Now, let’s create the environment-specific custom files.

Let’s create the Kustomize file for the development environment, along with the secret generator to read the environment variables.

First, I’ll create the kustomization.yaml file to define the kustomize for our resources.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

namePrefix: dev-
nameSuffix: "-001"
namespace: dev

secretGenerator:
  - name: env-secret
    envs:
      - secret-dev-env.env

images:
- name: denizturkmen/python
  newTag: env1

As demonstrated above, environment-based abstraction is effectively achieved. Typically, patching methods like JSON 6902 and strategic merge patches are used for environments like DEVELOPMENT, UAT, STAGING and PRODUCTION. However, these methods limit the level of abstraction and are not significantly different from Helm templates. Instead, we should consolidate all processes into a single Kustomize file. In this approach, environment variables are read from a secretGenerator, and image changes are handled by updating tags with the GitLab image tag.

Next, I’ll create the secret-dev-env.env file to establish the secretGenerator for our resources.

MY_VARIABLE="This is DEV environment....."

Let’s push this to gitlab, but first let’s switch to the relevant branch.

git checkout -b development

git add .

git commit -m "Adding kustomize manifest for development"

git push origin

Let’s create the Kustomize file for the uat environment, along with the secret generator to read the environment variables.

First, I’ll create the kustomization.yaml file to define the kustomize for our resources.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

namePrefix: uat-
nameSuffix: "-001"
namespace: uat

secretGenerator:
  - name: env-secret
    envs:
      - secret-uat-env.env

images:
- name: denizturkmen/python
  newTag: env1

Next, I’ll create the secret-uat-env.env file to establish the secretGenerator for our resources.

MY_VARIABLE="This is UAT environment....."

Let’s push this to gitlab, but first let’s switch to the relevant branch.

git checkout -b uat

git add .

git commit -m "Adding kustomize manifest for uat"

git push origin

Let’s create the Kustomize file for the staging environment, along with the secret generator to read the environment variables.

First, I’ll create the kustomization.yaml file to define the kustomize for our resources.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

namePrefix: staging-
nameSuffix: "-001"
namespace: staging

secretGenerator:
  - name: env-secret
    envs:
      - secret-staging-env.env

images:
- name: denizturkmen/python
  newTag: env1

Next, I’ll create the secret-staging-env.env file to establish the secretGenerator for our resources.

MY_VARIABLE="This is STAGING environment....."

Let’s push this to gitlab, but first let’s switch to the relevant branch.

git checkout -b staging

git add .

git commit -m "Adding kustomize manifest for staging"

git push origin

Let’s create the Kustomize file for the production environment, along with the secret generator to read the environment variables.

First, I’ll create the kustomization.yaml file to define the kustomize for our resources.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../base

namePrefix: production-
nameSuffix: "-001"
namespace: production

secretGenerator:
  - name: env-secret
    envs:
      - secret-production-env.env

images:
- name: denizturkmen/python
  newTag: env1

Next, I’ll create the secret-production-env.env file to establish the secretGenerator for our resources.

MY_VARIABLE="This is PRODUCTION environment....."

Let’s push this to gitlab, but first let’s switch to the relevant branch.

git checkout -b production

git add .

git commit -m "Adding kustomize manifest for production"

git push origin

Now it’s time to configure the CI pipeline in Gitlab and trigger ArgoCD through Gitlab.

Our gitlab-ci.yml file will consist of 5 stages. These are,

• SonarCheck
• Build for Dockerfile
• Push New Tag
• Deployment with ArgoCD
• HealtyCheck and Rollback

Before moving on to our example, let’s look at our branches in both the gitops repository and the project repository.

Gitops Repository

Python Example Repository

Now, let’s explain step by step. First, let’s explain the Sonarqube check step.

Sonarqube Check:
  stage: SonarqubeCheck
  image: 
    name: sonarsource/sonar-scanner-cli:latest
    entrypoint: [""]
  variables:
    SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"  # Defines the location of the analysis task cache
    GIT_DEPTH: "0"  # Tells git to fetch all the branches of the project, required by the analysis task
  cache:
    key: "${CI_JOB_NAME}"
    paths:
      - .sonar/cache
  script: 
    - sonar-scanner
  allow_failure: true
  tags:
    - devops-shell-1
  only:
    - master
    - development
    - uat
    - staging
    - production

You can add it by following the steps in the Sonarqube UI. I won’t go into much detail for the Sonarqube step. You can see the result output below

I planned the CI and CD pipeline as follows: When a pipeline is triggered in the development branch of the project, I configured the CI to update the newtag value in the kustomization.yaml file of the development branch in the GitOps repository with the new image. Similarly, for uat, it updates the uat branch, for staging, it updates the staging branch, and for production, it updates the production branch. We don't have any actions for branches outside of these, as I have designed it for four environments, with the pipeline building and deploying to the corresponding environment for each.

Next, Now, let’s take a look at the Docker build step.

Build for Dockerfile:
  stage: Build
  before_script:
    - docker login -u $username -p $password
    - echo $CI_PIPELINE_IID
    - echo $IMAGE_ID
  tags:
    - devops-shell-1
  script:
    - docker image build --tag denizturkmen/python:$IMAGE_ID --tag denizturkmen/python:latest .
    - docker push denizturkmen/python:$IMAGE_ID
    - docker push denizturkmen/python:latest

I’m pushing the image to the public Docker Hub. To do this, in the ‘before_script’ script, we write the command that will run when the pipeline first starts. This command is what we log into Docker Hub. In the other steps, we tag the image and push it to Docker Hub. Here, we read the $ variables from the GitLab CI/CD variable field.

CI&CD variable list

Now, let’s take a look at the step that forces the kustomization to the development branch. However, the same ‘gitlab-ci.yml’ file is present in the other branches as well.

Push NewTag&NewImage for Kustomization:
  stage: PushNewTag
  tags:
    - devops-shell-1
  script: |
      echo $CI_COMMIT_BRANCH; 
      echo "-----------------------------------";
      echo $CI_COMMIT_REF_NAME;

      if [ "$CI_COMMIT_REF_NAME" == "development" ]; then
        echo "This is the development branch.";
        echo "New $IMAGE_ID image customization yaml newtag is being pushed..... "
        git clone https://gitlab_username:$GITLAB_TOKEN@gitlab.devops-deniz.net/gitops-deployment/gitops.git
        cd gitops/
        git checkout development
        git pull origin development 
        cd PythonEx/kustomize/overlays/development
        ls -ltr
        cat kustomization.yaml
        sed -i "s|^\(\s*newTag:\s*\).*|\1${IMAGE_ID}|g" kustomization.yaml
        cat kustomization.yaml
        echo " "
        git add kustomization.yaml
        git commit -m "Update image tag to ${IMAGE_ID}"
        git push --force origin development
      
      elif [ "$CI_COMMIT_REF_NAME" == "uat" ]; then
        echo "This is the UAT branch.";
        echo "New $IMAGE_ID image customization yaml newtag is being pushed..... "
        git clone https://gitlab_username:$GITLAB_TOKEN@gitlab.devops-deniz.net/gitops-deployment/gitops.git
        cd gitops/
        git checkout uat
        git pull origin uat 
        cd PythonEx/kustomize/overlays/uat
        ls -ltr
        cat kustomization.yaml
        sed -i "s|^\(\s*newTag:\s*\).*|\1${IMAGE_ID}|g" kustomization.yaml
        cat kustomization.yaml
        echo " "
        git add kustomization.yaml
        git commit -m "Update image tag to ${IMAGE_ID}"
        git push --force origin uat
      
      elif [ "$CI_COMMIT_REF_NAME" == "staging" ]; then
        echo "This is the STAGING branch.";
        echo "New $IMAGE_ID image customization yaml newtag is being pushed..... "
        git clone https://gitlab_username:$GITLAB_TOKEN@gitlab.devops-deniz.net/gitops-deployment/gitops.git
        cd gitops
        git checkout staging
        git pull origin staging 
        cd PythonEx/kustomize/overlays/staging
        ls -ltr
        cat kustomization.yaml
        sed -i "s|^\(\s*newTag:\s*\).*|\1${IMAGE_ID}|g" kustomization.yaml
        cat kustomization.yaml
        echo " "
        git add kustomization.yaml
        git commit -m "Update image tag to ${IMAGE_ID}"
        git push --force origin staging
      
      elif [ "$CI_COMMIT_REF_NAME" == "production" ]; then
        echo "This is the PRODUCTION branch.";
        echo "New $IMAGE_ID image customization yaml newtag is being pushed..... "
        git clone https://gitlab_username:$GITLAB_TOKEN@gitlab.devops-deniz.net/gitops-deployment/gitops.git
        cd gitops
        git checkout production
        git pull origin production 
        cd PythonEx/kustomize/overlays/production
        ls -ltr
        cat kustomization.yaml
        sed -i "s|^\(\s*newTag:\s*\).*|\1${IMAGE_ID}|g" kustomization.yaml
        cat kustomization.yaml
        echo " "
        git add kustomization.yaml
        git commit -m "Update image tag to ${IMAGE_ID}"
        git push --force origin production
      
      else
        echo "Branch not selected for distribution...."
      fi

To explain the above output a bit.

• We keep our kustomization definitions in the GitOps repository, so when we get a new build, we need to update the newtag field in the relevant branch with this new build.
• Create a Personal Access Token (PAT)

Navigate to Gitlab:

Go to your Gitlab instance and sign in.

Go to your profile settings:

• Click on your avatar in the top-right corner and select “Edit Profile”.

Access Personal Access Tokens:

• On the left sidebar, click on “Access Tokens”.

Create a new token:

• Give your token a name (e.g., CI Clone Token).
• Set an expiration date if needed.
• Select the necessary scopes. For cloning repositories, you need the read_repository scope.
• Click “Create personal access token” and save the generated token securely.

Add the Token as a CI/CD Variable: Go to your project’s settings:

• In GitLab, navigate to your project.
• Access CI/CD settings:
• On the left sidebar, click on “Settings” > “CI/CD”.

Add the token as a variable:

• Scroll down to the “Variables” section and click “Expand”.
• Click “Add Variable”.
• Name the variable (e.g., GITLAB_TOKEN).
• Paste your token into the “Value” field.
• Set the type as “Masked” for security.
• Click “Save variables”.

Gitlab-Token Variable

• Next, What I’m trying to do in the if-else block is this: If a build starts from the branch where I want to deploy, I wrote the pipeline to commit to the corresponding branch in the GitOps repository. First, I switch to the branch where the commit will be made using the checkout command, and if there are any existing commits, I pull them. Then, I update the newtag value in the kustomization.yaml file in the relevant branch with the new build. I do this using the sed editor, but you can also use jq, yq, or kustomize; however, these three must be installed on the GitLab runner. When I update the newtag value, I commit it with the build number so that it is easier to track in the Git repository and to find it by commit ID when needed (e.g., $IMAGE_ID).
• The most important step was to identify the variable and implement a solution. Let’s look at that now.

variables:
  IMAGE_ID: $CI_COMMIT_SHORT_SHA-$CI_PIPELINE_IID

The reason I did this is as follows: A developer might want to trigger a new build without committing to the relevant branch. However, since no new commit is made, both the GitLab runner and Docker will use the same CI_COMMIT_SHORT_SHA (commit ID) from the cache. When checking the Git status, the pipeline will fail because there is no new commit. To prevent this, I added CI_PIPELINE_IID at the end, ensuring that each build generates a new image and successfully avoiding this issue.

If I had continued using CI_COMMIT_SHORT_SHA without generating my own IMAGE_ID, I would have encountered the error below. To resolve this, I created an IMAGE_ID and pushed it to Docker Hub.

Fail job

The output of a successful job is shown below.

Success job

Next. Now, let’s explain the stage where we deploy to the relevant branch. “gitlab-ci.yml” for development branch.

Deployment for Development Envrionment:
  stage: Deployment
  tags:
    - devops-shell-1
  script:
    - echo "Argocd Deployment is starting for Development Environment"
    - argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
    - argocd app get argocd/gitops-arch-development-1 --hard-refresh
    - sleep 30
    - argocd app sync --grpc-web argocd/gitops-arch-development-1
  when: manual

Since I set the ArgoCD policy to manual, we first run the argocd login command. argocd login ip:port --username user --password pass. Here, we enter the Kubernetes IP and the NodePort IP where ArgoCD is installed, along with the username and password.Then, I perform a hard refresh on the application and introduce a delay. The reason for the delay is to prevent the pipeline from failing due to issues like network problems in the ArgoCD application. Finally, we run the sync command to align the GitOps repository with ArgoCD.

The output of a successful job is shown below.

If we look at the result in the Argo-CD UI……

Next. Below is the application.yaml file for the development branch that we deployed.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: gitops-arch-1-dev
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: dev
    server: https://kubernetes.default.svc
  project: default
  source:
    path: PythonEx/kustomize/overlays/development
    repoURL: https://gitlab.devops-deniz.net/gitops-deployment/gitops.git
    targetRevision: development
  syncPolicy:
    automated: null
    syncOptions:
      - Validate=true
      - CreateNamespace=false
      - PrunePropagationPolicy=foreground
      - PruneLast=true

Finally, let’s write a job for health checks and rollback.

HealtyCheck and Rollback for Development Envrionment:
  stage: HealtyRollback
  tags:
    - devops-shell-1
  needs:
    - Deployment for Development Envrionment
  script: | 
      argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
      sleep 60
      
      echo "Checking health status of the ArgoCD application..."
           syncOutput=$(argocd app get argocd/gitops-arch-development-1 --grpc-web --output json | 
      jq -r '.status.resources[] | select(.kind=="Deployment") | .health.status')
      
      echo $syncOutput
      
      if [[ "$syncOutput" != "Healthy" ]]; then
        argocd app get --grpc-web argocd/gitops-arch-development-1
        argocd app rollback argocd/gitops-arch-development-1 
        sleep 5
        echo "Sync failed. Initiating rollback of last revision..."
        exit 1  # Force rollback stage if validation fails
     
      else
        echo 

To explain the job a bit further: since the app does not change its state immediately, we introduce a 60-second delay to ensure we capture the correct state. After the delay, we check the app’s health status. If the status is anything other than ‘Healthy’ — such as ‘Progressing,’ ‘Failed,’ etc… A command automatically triggers to roll back to the previous running state.

When the deployment is successful, the health check and rollback job sync are displayed as output, indicating that they completed successfully.

Success job….

When the job encounters an error, it waits for 60 seconds. If there is any issue other than the API health, the previous worker performs a release rollback, and I terminate the pipeline to indicate that something went wrong.

Fail job…..

If we look at the result in the Argo-CD UI……

We see an ‘OutOfSync’ status because there is no synchronization between ArgoCD and the GitOps repository.

Note: Remember that we have set the ArgoCD sync policy to manual!

You can see the .gitlab-ci.yml file containing all the jobs below.

stages:
  - SonarqubeCheck
  - Build
  - PushNewTag
  - Deployment
  - HealtyRollback

variables:
  IMAGE_ID: $CI_COMMIT_SHORT_SHA-$CI_PIPELINE_IID
  
Sonarqube Check:
  stage: SonarqubeCheck
  image: 
    name: sonarsource/sonar-scanner-cli:latest
    entrypoint: [""]
  variables:
    SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"  # Defines the location of the analysis task cache
    GIT_DEPTH: "0"  # Tells git to fetch all the branches of the project, required by the analysis task
  cache:
    key: "${CI_JOB_NAME}"
    paths:
      - .sonar/cache
  script: 
    - sonar-scanner
  allow_failure: true
  tags:
    - devops-shell-1
  only:
    - master
    - development
    - uat
    - staging
    - production


Build for Dockerfile:
  stage: Build
  before_script:
    - docker login -u $username -p $password
    - echo $CI_PIPELINE_IID
    - echo $IMAGE_ID
  tags:
    - devops-shell-1
  script:
    - docker image build --tag denizturkmen/python:$IMAGE_ID --tag denizturkmen/python:latest .
    - docker push denizturkmen/python:$IMAGE_ID
    - docker push denizturkmen/python:latest

Push NewTag&NewImage for Kustomization:
  stage: PushNewTag
  tags:
    - devops-shell-1
  script: |
      echo $CI_COMMIT_BRANCH; 
      echo "-----------------------------";
      if [ "$CI_COMMIT_REF_NAME" == "development" ]; then
        echo "This is the development branch.";
        echo "New $IMAGE_ID image customization yaml newtag is being pushed..... "
        git clone https://gitlab_username:$GITLAB_TOKEN@gitlab.devops-deniz.net/gitops-deployment/gitops.git
        cd gitops/
        git checkout development
        git pull origin development 
        cd PythonEx/kustomize/overlays/development
        ls -ltr
        cat kustomization.yaml
        sed -i "s|^\(\s*newTag:\s*\).*|\1${IMAGE_ID}|g" kustomization.yaml
        cat kustomization.yaml
        echo " "
        git add kustomization.yaml
        git commit -m "Update image tag to ${IMAGE_ID}"
        git push --force origin development
      
      elif [ "$CI_COMMIT_REF_NAME" == "uat" ]; then
        echo "This is the UAT branch.";
        echo "New $IMAGE_ID image customization yaml newtag is being pushed..... "
        git clone https://gitlab_username:$GITLAB_TOKEN@gitlab.devops-deniz.net/gitops-deployment/gitops.git
        cd gitops/
        git checkout uat
        git pull origin uat 
        cd PythonEx/kustomize/overlays/uat
        ls -ltr
        cat kustomization.yaml
        sed -i "s|^\(\s*newTag:\s*\).*|\1${IMAGE_ID}|g" kustomization.yaml
        cat kustomization.yaml
        echo " "
        git add kustomization.yaml
        git commit -m "Update image tag to ${IMAGE_ID}"
        git push --force origin uat
      
      elif [ "$CI_COMMIT_REF_NAME" == "staging" ]; then
        echo "This is the STAGING branch.";
        echo "New $IMAGE_ID image customization yaml newtag is being pushed..... "
        git clone https://gitlab_username:$GITLAB_TOKEN@gitlab.devops-deniz.net/gitops-deployment/gitops.git
        cd gitops
        git checkout staging
        git pull origin staging 
        cd PythonEx/kustomize/overlays/staging
        ls -ltr
        cat kustomization.yaml
        sed -i "s|^\(\s*newTag:\s*\).*|\1${IMAGE_ID}|g" kustomization.yaml
        cat kustomization.yaml
        echo " "
        git add kustomization.yaml
        git commit -m "Update image tag to ${IMAGE_ID}"
        git push --force origin staging
      
      elif [ "$CI_COMMIT_REF_NAME" == "production" ]; then
        echo "This is the PRODUCTION branch.";
        echo "New $IMAGE_ID image customization yaml newtag is being pushed..... "
        git clone https://gitlab_username:$GITLAB_TOKEN@gitlab.devops-deniz.net/gitops-deployment/gitops.git
        cd gitops
        git checkout production
        git pull origin production 
        cd PythonEx/kustomize/overlays/production
        ls -ltr
        cat kustomization.yaml
        sed -i "s|^\(\s*newTag:\s*\).*|\1${IMAGE_ID}|g" kustomization.yaml
        cat kustomization.yaml
        echo " "
        git add kustomization.yaml
        git commit -m "Update image tag to ${IMAGE_ID}"
        git push --force origin production
      
      else
        echo "Branch not selected for distribution...."
      fi


Deployment for Development Envrionment:
  stage: Deployment
  tags:
    - devops-shell-1
  script:
    - echo "Argocd Deployment is starting for Development Environment"
    - argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
    - argocd app get argocd/gitops-arch-development-1 --hard-refresh
    - sleep 5
    - argocd app sync --grpc-web argocd/gitops-arch-development-1
  when: manual

HealtyCheck and Rollback for Development Envrionment:
  stage: HealtyRollback
  tags:
    - devops-shell-1
  needs:
    - Deployment for Development Envrionment
  script: | 
      argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
      sleep 60
      
      echo "Checking health status of the ArgoCD application..."
           syncOutput=$(argocd app get argocd/gitops-arch-development-1 --grpc-web --output json | 
      jq -r '.status.resources[] | select(.kind=="Deployment") | .health.status')
      
      echo $syncOutput
      
      if [[ "$syncOutput" != "Healthy" ]]; then
        argocd app get --grpc-web argocd/gitops-arch-development-1
        argocd app rollback argocd/gitops-arch-development-1 
        sleep 5
        echo "Sync failed. Initiating rollback of last revision..."
        exit 1  # Force rollback stage if validation fails
     
      else
        echo "Sync was successful. New release is successful."
      fi

Note: The same processes will apply to other branches (UAT, staging, production), so I’m not including them in the article.

To explain the first architecture briefly: the design I created allows for branch-based deployment. For example, the development branch in the project repository tracks the development branch in the GitOps repository. My aim was to enable branch-to-branch deployment — using the UAT GitOps repository for UAT, the staging GitOps repository for staging, and the production GitOps repository for production. However, this structure somewhat contradicts the custom setup. I’ll leave it to your discretion.

The purpose of the second architecture is to allow a build to be triggered from any branch in the project and deploy to any desired environment. Here, a build can start from any branch, but it will only track the master branch of the GitOps repository, as ArgoCD listens to a single branch.

Now, let’s explain step by step. First, let’s explain the Sonarqube check step.

Sonarqube Check:
  stage: SonarqubeCheck
  image: 
    name: sonarsource/sonar-scanner-cli:latest
    entrypoint: [""]
  variables:
    SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"  # Defines the location of the analysis task cache
    GIT_DEPTH: "0"  # Tells git to fetch all the branches of the project, required by the analysis task
  cache:
    key: "${CI_JOB_NAME}"
    paths:
      - .sonar/cache
  script: 
    - sonar-scanner
  allow_failure: true
  tags:
    - devops-shell-1
  only:
    - master
    - development
    - uat
    - staging
    - production

You can add it by following the steps in the Sonarqube UI. I won’t go into much detail for the Sonarqube step. You can see the result output below

Sonarqube UI

In the build step, we create a Docker image from the Dockerfile and push it to Docker Hub, which serves as our private registry.

Build for Dockerfile:
  stage: Build
  before_script:
    - docker login -u $username -p $password
    - echo $CI_PIPELINE_IID
    - echo $IMAGE_ID
  tags:
    - devops-shell-1
  script:
    - docker image build --tag denizturkmen/python:$IMAGE_ID --tag denizturkmen/python:latest .
    - docker push denizturkmen/python:$IMAGE_ID
    - docker push denizturkmen/python:latest

In the PushNewTag phase, My plan is as follows: when we run a job from the Python example repository, which is our project repository, the newly created image is pushed to all environments under the overlays in the master branch of the GitOps repository. This way, if the Python example branch in our project repository runs, a new image will be created, and the relevant overlays in the master branch of the GitOps repository will be committed.

Push NewTag&NewImage for Kustomization:
  stage: PushNewTag
  tags:
    - devops-shell-1
  script: |
      echo "This is the development branch.";
      echo "New $IMAGE_ID image customization yaml newtag is being pushed..... "
      git clone https://gitlab_username:$GITLAB_TOKEN@gitlab.devops-deniz.net/gitops-deployment/gitops.git
      git checkout master
      git pull origin master 
      cd gitops/PythonEx/kustomize/overlays/development
      ls -ltr
      cat kustomization.yaml
      sed -i "s|^\(\s*newTag:\s*\).*|\1${IMAGE_ID}|g" kustomization.yaml
      cat kustomization.yaml
      echo " "
      git add kustomization.yaml
      git commit -m "Update image tag to ${IMAGE_ID}"
      git push --force origin master
      cd -
      pwd
      ls -ltr

      echo "This is the uat branch.";
      echo "New $IMAGE_ID image customization yaml newtag is being pushed..... "
      git checkout master
      git pull origin master 
      cd gitops/PythonEx/kustomize/overlays/uat
      ls -ltr
      cat kustomization.yaml
      sed -i "s|^\(\s*newTag:\s*\).*|\1${IMAGE_ID}|g" kustomization.yaml
      cat kustomization.yaml
      echo " "
      git add kustomization.yaml
      git commit -m "Update image tag to ${IMAGE_ID}"
      git push --force origin master
      cd - 

      echo "This is the staging branch.";
      echo "New $IMAGE_ID image customization yaml newtag is being pushed..... "
      git checkout master
      git pull origin master 
      cd gitops/PythonEx/kustomize/overlays/staging
      ls -ltr
      cat kustomization.yaml
      sed -i "s|^\(\s*newTag:\s*\).*|\1${IMAGE_ID}|g" kustomization.yaml
      cat kustomization.yaml
      echo " "
      git add kustomization.yaml
      git commit -m "Update image tag to ${IMAGE_ID}"
      git push --force origin master
      cd -   
      
      echo "This is the production branch.";
      echo "New $IMAGE_ID image customization yaml newtag is being pushed..... "
      git checkout master
      git pull origin master 
      cd gitops/PythonEx/kustomize/overlays/production
      ls -ltr
      cat kustomization.yaml
      sed -i "s|^\(\s*newTag:\s*\).*|\1${IMAGE_ID}|g" kustomization.yaml
      cat kustomization.yaml
      echo " "
      git add kustomization.yaml
      git commit -m "Update image tag to ${IMAGE_ID}"
      git push --force origin master
      cd -

Below is the output of the commit to the master branch of the GitOps repository.

The output of a successful job is shown below.

Success Job…

The relevant jobs for environment-based deployment are listed below. To explain briefly: the job logs into ArgoCD, runs the sync command, and then triggers the health check and rollback job using the needs keyword.

Deployment for Development Envrionment:
  stage: Deployment
  tags:
    - devops-shell-1
  script:
    - echo "Argocd Deployment is starting for Development Environment"
    - argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
    - argocd app get argocd/gitops-arch-development-2 --hard-refresh
    - sleep 5
    - argocd app sync --grpc-web argocd/gitops-arch-development-2
  when: manual

HealtyCheck and Rollback for Development Envrionment:
  stage: HealtyRollback
  tags:
    - devops-shell-1
  needs:
    - Deployment for Development Envrionment
  script: | 
      argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
      sleep 60
      
      echo "Checking health status of the ArgoCD application..."
           syncOutput=$(argocd app get argocd/gitops-arch-development-2 --grpc-web --output json | 
      jq -r '.status.resources[] | select(.kind=="Deployment") | .health.status')
      
      echo $syncOutput
      
      if [[ "$syncOutput" != "Healthy" ]]; then
        argocd app get --grpc-web argocd/gitops-arch-development-2
        argocd app rollback argocd/gitops-arch-development-2 
        sleep 5
        echo "Sync failed. Initiating rollback of last revision..."
        exit 1  # Force rollback stage if validation fails
     
      else
        echo "Sync was successful. New release is successful."
      fi

Deployment for Uat Envrionment:
  stage: Deployment
  tags:
    - devops-shell-1
  script:
    - echo "Argocd Deployment is starting for uat Environment"
    - argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
    - argocd app get argocd/gitops-arch-uat-2 --hard-refresh
    - sleep 5
    - argocd app sync --grpc-web argocd/gitops-arch-uat-2
  when: manual

HealtyCheck and Rollback for Uat Envrionment:
  stage: HealtyRollback
  tags:
    - devops-shell-1
  needs:
    - Deployment for Uat Envrionment
  script: | 
      argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
      sleep 60
      
      echo "Checking health status of the ArgoCD application..."
           syncOutput=$(argocd app get argocd/gitops-arch-uat-2 --grpc-web --output json | 
      jq -r '.status.resources[] | select(.kind=="Deployment") | .health.status')
      
      echo $syncOutput
      
      if [[ "$syncOutput" != "Healthy" ]]; then
        argocd app get --grpc-web argocd/gitops-arch-uat-2
        argocd app rollback argocd/gitops-arch-uat-2 
        sleep 5
        echo "Sync failed. Initiating rollback of last revision..."
        exit 1  # Force rollback stage if validation fails
     
      else
        echo "Sync was successful. New release is successful."
      fi

Deployment for Staging Envrionment:
  stage: Deployment
  tags:
    - devops-shell-1
  script:
    - echo "Argocd Deployment is starting for staging Environment"
    - argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
    - argocd app get argocd/gitops-arch-staging-2 --hard-refresh
    - sleep 5
    - argocd app sync --grpc-web argocd/gitops-arch-staging-2
  when: manual

HealtyCheck and Rollback for Staging Envrionment:
  stage: HealtyRollback
  tags:
    - devops-shell-1
  needs:
    - Deployment for Staging Envrionment
  script: | 
      argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
      sleep 60
      
      echo "Checking health status of the ArgoCD application..."
           syncOutput=$(argocd app get argocd/gitops-arch-staging-2 --grpc-web --output json | 
      jq -r '.status.resources[] | select(.kind=="Deployment") | .health.status')
      
      echo $syncOutput
      
      if [[ "$syncOutput" != "Healthy" ]]; then
        argocd app get --grpc-web argocd/gitops-arch-staging-2
        argocd app rollback argocd/gitops-arch-staging-2 
        sleep 5
        echo "Sync failed. Initiating rollback of last revision..."
        exit 1  # Force rollback stage if validation fails
     
      else
        echo "Sync was successful. New release is successful."
      fi

Deployment for Production Envrionment:
  stage: Deployment
  tags:
    - devops-shell-1
  script:
    - echo "Argocd Deployment is starting for production Environment"
    - argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
    - argocd app get argocd/gitops-arch-production-2 --hard-refresh
    - sleep 5
    - argocd app sync --grpc-web argocd/gitops-arch-production-1
  when: manual

HealtyCheck and Rollback for Production Envrionment:
  stage: HealtyRollback
  tags:
    - devops-shell-1
  needs:
    - Deployment for Production Envrionment
  script: | 
      argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
      sleep 60
      
      echo "Checking health status of the ArgoCD application..."
           syncOutput=$(argocd app get argocd/gitops-arch-production-2 --grpc-web --output json | 
      jq -r '.status.resources[] | select(.kind=="Deployment") | .health.status')
      
      echo $syncOutput
      
      if [[ "$syncOutput" != "Healthy" ]]; then
        argocd app get --grpc-web argocd/gitops-arch-production-2
        argocd app rollback argocd/gitops-arch-production-2 
        sleep 5
        echo "Sync failed. Initiating rollback of last revision..."
        exit 1  # Force rollback stage if validation fails
     
      else
        echo "Sync was successful. New release is successful."
      fi

You can see the .gitlab-ci.yml file containing all the jobs below.

stages:
  - SonarqubeCheck
  - Build
  - PushNewTag
  - Deployment
  - HealtyRollback

variables:
  IMAGE_ID: $CI_COMMIT_SHORT_SHA-$CI_PIPELINE_IID
  
Sonarqube Check:
  stage: SonarqubeCheck
  image: 
    name: sonarsource/sonar-scanner-cli:latest
    entrypoint: [""]
  variables:
    SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"  # Defines the location of the analysis task cache
    GIT_DEPTH: "0"  # Tells git to fetch all the branches of the project, required by the analysis task
  cache:
    key: "${CI_JOB_NAME}"
    paths:
      - .sonar/cache
  script: 
    - sonar-scanner
  allow_failure: true
  tags:
    - devops-shell-1
  only:
    - master
    - development
    - uat
    - staging
    - production


Build for Dockerfile:
  stage: Build
  before_script:
    - docker login -u $username -p $password
    - echo $CI_PIPELINE_IID
    - echo $IMAGE_ID
  tags:
    - devops-shell-1
  script:
    - docker image build --tag denizturkmen/python:$IMAGE_ID --tag denizturkmen/python:latest .
    - docker push denizturkmen/python:$IMAGE_ID
    - docker push denizturkmen/python:latest

Push NewTag&NewImage for Kustomization:
  stage: PushNewTag
  tags:
    - devops-shell-1
  script: |
      echo "This is the development branch.";
      echo "New $IMAGE_ID image customization yaml newtag is being pushed..... "
      git clone https://gitlab_username:$GITLAB_TOKEN@gitlab.devops-deniz.net/gitops-deployment/gitops.git
      git checkout master
      git pull origin master 
      cd gitops/PythonEx/kustomize/overlays/development
      ls -ltr
      cat kustomization.yaml
      sed -i "s|^\(\s*newTag:\s*\).*|\1${IMAGE_ID}|g" kustomization.yaml
      cat kustomization.yaml
      echo " "
      git add kustomization.yaml
      git commit -m "Update image tag to ${IMAGE_ID}"
      git push --force origin master
      cd -
      pwd
      ls -ltr

      echo "This is the uat branch.";
      echo "New $IMAGE_ID image customization yaml newtag is being pushed..... "
      git checkout master
      git pull origin master 
      cd gitops/PythonEx/kustomize/overlays/uat
      ls -ltr
      cat kustomization.yaml
      sed -i "s|^\(\s*newTag:\s*\).*|\1${IMAGE_ID}|g" kustomization.yaml
      cat kustomization.yaml
      echo " "
      git add kustomization.yaml
      git commit -m "Update image tag to ${IMAGE_ID}"
      git push --force origin master
      cd - 

      echo "This is the staging branch.";
      echo "New $IMAGE_ID image customization yaml newtag is being pushed..... "
      git checkout master
      git pull origin master 
      cd gitops/PythonEx/kustomize/overlays/staging
      ls -ltr
      cat kustomization.yaml
      sed -i "s|^\(\s*newTag:\s*\).*|\1${IMAGE_ID}|g" kustomization.yaml
      cat kustomization.yaml
      echo " "
      git add kustomization.yaml
      git commit -m "Update image tag to ${IMAGE_ID}"
      git push --force origin master
      cd -   
      
      echo "This is the production branch.";
      echo "New $IMAGE_ID image customization yaml newtag is being pushed..... "
      git checkout master
      git pull origin master 
      cd gitops/PythonEx/kustomize/overlays/production
      ls -ltr
      cat kustomization.yaml
      sed -i "s|^\(\s*newTag:\s*\).*|\1${IMAGE_ID}|g" kustomization.yaml
      cat kustomization.yaml
      echo " "
      git add kustomization.yaml
      git commit -m "Update image tag to ${IMAGE_ID}"
      git push --force origin master
      cd - 

Deployment for Development Envrionment:
  stage: Deployment
  tags:
    - devops-shell-1
  script:
    - echo "Argocd Deployment is starting for Development Environment"
    - argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
    - argocd app get argocd/gitops-arch-development-2 --hard-refresh
    - sleep 5
    - argocd app sync --grpc-web argocd/gitops-arch-development-2
  when: manual

HealtyCheck and Rollback for Development Envrionment:
  stage: HealtyRollback
  tags:
    - devops-shell-1
  needs:
    - Deployment for Development Envrionment
  script: | 
      argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
      sleep 60
      
      echo "Checking health status of the ArgoCD application..."
           syncOutput=$(argocd app get argocd/gitops-arch-development-2 --grpc-web --output json | 
      jq -r '.status.resources[] | select(.kind=="Deployment") | .health.status')
      
      echo $syncOutput
      
      if [[ "$syncOutput" != "Healthy" ]]; then
        argocd app get --grpc-web argocd/gitops-arch-development-2
        argocd app rollback argocd/gitops-arch-development-2 
        sleep 5
        echo "Sync failed. Initiating rollback of last revision..."
        exit 1  # Force rollback stage if validation fails
     
      else
        echo "Sync was successful. New release is successful."
      fi

Deployment for Uat Envrionment:
  stage: Deployment
  tags:
    - devops-shell-1
  script:
    - echo "Argocd Deployment is starting for uat Environment"
    - argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
    - argocd app get argocd/gitops-arch-uat-2 --hard-refresh
    - sleep 5
    - argocd app sync --grpc-web argocd/gitops-arch-uat-2
  when: manual

HealtyCheck and Rollback for Uat Envrionment:
  stage: HealtyRollback
  tags:
    - devops-shell-1
  needs:
    - Deployment for Uat Envrionment
  script: | 
      argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
      sleep 60
      
      echo "Checking health status of the ArgoCD application..."
           syncOutput=$(argocd app get argocd/gitops-arch-uat-2 --grpc-web --output json | 
      jq -r '.status.resources[] | select(.kind=="Deployment") | .health.status')
      
      echo $syncOutput
      
      if [[ "$syncOutput" != "Healthy" ]]; then
        argocd app get --grpc-web argocd/gitops-arch-uat-2
        argocd app rollback argocd/gitops-arch-uat-2 
        sleep 5
        echo "Sync failed. Initiating rollback of last revision..."
        exit 1  # Force rollback stage if validation fails
     
      else
        echo "Sync was successful. New release is successful."
      fi

Deployment for Staging Envrionment:
  stage: Deployment
  tags:
    - devops-shell-1
  script:
    - echo "Argocd Deployment is starting for staging Environment"
    - argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
    - argocd app get argocd/gitops-arch-staging-2 --hard-refresh
    - sleep 5
    - argocd app sync --grpc-web argocd/gitops-arch-staging-2
  when: manual

HealtyCheck and Rollback for Staging Envrionment:
  stage: HealtyRollback
  tags:
    - devops-shell-1
  needs:
    - Deployment for Staging Envrionment
  script: | 
      argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
      sleep 60
      
      echo "Checking health status of the ArgoCD application..."
           syncOutput=$(argocd app get argocd/gitops-arch-staging-2 --grpc-web --output json | 
      jq -r '.status.resources[] | select(.kind=="Deployment") | .health.status')
      
      echo $syncOutput
      
      if [[ "$syncOutput" != "Healthy" ]]; then
        argocd app get --grpc-web argocd/gitops-arch-staging-2
        argocd app rollback argocd/gitops-arch-staging-2 
        sleep 5
        echo "Sync failed. Initiating rollback of last revision..."
        exit 1  # Force rollback stage if validation fails
     
      else
        echo "Sync was successful. New release is successful."
      fi

Deployment for Production Envrionment:
  stage: Deployment
  tags:
    - devops-shell-1
  script:
    - echo "Argocd Deployment is starting for production Environment"
    - argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
    - argocd app get argocd/gitops-arch-production-2 --hard-refresh
    - sleep 5
    - argocd app sync --grpc-web argocd/gitops-arch-production-1
  when: manual

HealtyCheck and Rollback for Production Envrionment:
  stage: HealtyRollback
  tags:
    - devops-shell-1
  needs:
    - Deployment for Production Envrionment
  script: | 
      argocd login 192.168.1.7:31036 --username admin --password 21195831 --insecure
      sleep 60
      
      echo "Checking health status of the ArgoCD application..."
           syncOutput=$(argocd app get argocd/gitops-arch-production-2 --grpc-web --output json | 
      jq -r '.status.resources[] | select(.kind=="Deployment") | .health.status')
      
      echo $syncOutput
      
      if [[ "$syncOutput" != "Healthy" ]]; then
        argocd app get --grpc-web argocd/gitops-arch-production-2
        argocd app rollback argocd/gitops-arch-production-2 
        sleep 5
        echo "Sync failed. Initiating rollback of last revision..."
        exit 1  # Force rollback stage if validation fails
     
      else
        echo "Sync was successful. New release is successful."
      fi

To explain the architecture briefly: my goal is to push a new image to all environments regardless of which branch the pipeline starts from in the project repository. This allows developers to deploy to all environments as desired within a single pipeline flow.

The Results;

Let’s look at the output from argocd UI.

Conclusion

In this exploration of two distinct GitOps architectures, both approaches offer unique advantages for managing deployments across multiple environments. The first architecture provides branch-to-branch deployment, which aligns the project repository directly with its corresponding GitOps repository branch. This setup is particularly beneficial for those seeking to maintain environment-specific configurations and control, although it may introduce some complexity when integrating with custom structures.

The second architecture focuses on streamlining the deployment process by pushing new images to a single GitOps repository master branch, regardless of the originating project branch. This method simplifies the pipeline by allowing developers to manage deployments in a unified flow, enhancing ease of use and reducing operational overhead. It is especially advantageous for teams that prefer a straightforward and consolidated approach to managing deployments across environments.

Both architectures have their merits: the first offers precise control and alignment with GitOps principles, while the second emphasizes simplicity and developer efficiency. Ultimately, the choice depends on the specific needs and preferences of your development and operations teams. Balancing control with simplicity is key, and the best approach will be one that aligns with your team’s workflow, scalability requirements, and desired level of automation.

Referance;

DeNiZ Turkmen - Blog

Deniz Türkmen

Deniz Türkmen — Turkey | Professional Profile | LinkedIn

denizturkmen — Overview