# Building a Self-Healing GitOps Based Micro-services Platform on GKE with Argo CD, HPA & n8n
* * *
**Repository Link:** [https://github.com/sachindumalshan/gitops-repo.git](https://github.com/sachindumalshan/gitops-repo.git)
## Architecture Overview
Here is the system level architecture.
```plaintext
Google Cloud (GKE)
│
├── Kubernetes Cluster (gitops-cluster)
│ ├── default namespace
│ │ ├──── service-a (Device Service)
│ │ │ ├─ app.py
│ │ │ ├─ deployment.yaml
│ │ │ ├─ hpa.yaml
│ │ │ ├─ Dockerfile
│ │ │ └─ service.yaml
│ │ ├──── service-b (Sensor Service)
│ │ │ ├─ app.py
│ │ │ ├─ deployment.yaml
│ │ │ ├─ hpa.yaml
│ │ │ ├─ Dockerfile
│ │ │ └─ service.yaml
│ │ └──── service-c (Alert Service)
│ │ ├─ app.py
│ │ ├─ deployment.yaml
│ │ ├─ hpa.yaml
│ │ ├─ Dockerfile
│ │ └─ service.yaml
│ │
│ ├── argocd namespace
│ │ ├── argocd-server
│ │ ├── argocd-repo-server
│ │ ├── argocd-application-controller
│ │ └── argocd-dex-server
│ │
│ └── automation namespace
│ └── n8n
│
├── Google Artifact Registry
│
└── GitHub (Source of Truth)
```
## End-to-End Flow
```plaintext
Developer pushes code → GitHub
↓
Argo CD detects change
↓
Argo CD syncs to GKE
↓
Pods deployed / updated
↓
Kubernetes handles:
- Self-healing
- Auto-scaling (HPA)
↓
n8n monitors services
↓
Slack alerts if failure
```
* * *
## PHASE 1 - Install gcloud CLI & Setup GKE
![](https://cloudmate-test.s3.us-east-1.amazonaws.com/uploads/covers/685b63b17e7d46cc40b0aa24/309f095f-8de7-48da-bc44-66ea2015ec75.png align="center")
Update System
```bash
sudo apt-get update
```
Install required packages:
```bash
sudo apt-get install apt-transport-https ca-certificates gnupg curl
```
Import Google Cloud Public Key
* For newer distributions:
```bash
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg
```
* For older distributions:
```bash
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
```
* If unsupported:
```bash
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
```
Add Repository
* Newer systems:
```bash
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
```
* Older systems:
```bash
echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
```
Install gcloud
```bash
sudo apt-get update && sudo apt-get install google-cloud-cli
```
Login & Configure Project
```bash
gcloud auth login
gcloud projects list
gcloud config set project gitops-self-healing-7687
```
Enable APIs:
```bash
gcloud services enable \
container.googleapis.com \
compute.googleapis.com \
cloudbuild.googleapis.com \
artifactregistry.googleapis.com \
```
🟢 Create GKE Cluster
```bash
gcloud container clusters create gitops-cluster \
--zone asia-south1-a \
--num-nodes 2 \
--machine-type e2-medium
```
Get credentials:
```bash
gcloud container clusters get-credentials gitops-cluster \
--zone asia-south1-a
```
### ❗ ERROR 1 — GKE Auth Plugin Missing
If you see authentication errors:
Install plugin:
```bash
sudo apt-get install google-cloud-cli-gke-gcloud-auth-plugin
```
Verify:
```bash
which gke-gcloud-auth-plugin
```
Refresh kubeconfig:
```bash
gcloud container clusters get-credentials gitops-cluster \
--zone asia-south1-a
```
Test:
```bash
kubectl get nodes
```
![](https://cloudmate-test.s3.us-east-1.amazonaws.com/uploads/covers/685b63b17e7d46cc40b0aa24/e87eaba0-dc0d-476e-935a-fcb980c01005.png align="center")
* * *
## PHASE 2 - Build & Push Python Microservice
![](https://cloudmate-test.s3.us-east-1.amazonaws.com/uploads/covers/685b63b17e7d46cc40b0aa24/fba498cd-b5bf-4121-b70d-2b6ff7b77830.png align="center")
Sample Flask App: app.py
```python
from flask import Flask
import os
app = Flask(__name__)
@app.route("/health")
def health():
if os.getenv("FAIL") == "true":
return "FAIL", 500
return "OK", 200
@app.route("/")
def home():
return "Service A Running", 200
app.run(host="0.0.0.0", port=8080)
```
Dockerfile
```dockerfile
FROM python:3.11-slim
WORKDIR /app
RUN pip install flask
COPY app.py .
CMD ["python", "app.py"]
```
Create Artifact Registry
![](https://cloudmate-test.s3.us-east-1.amazonaws.com/uploads/covers/685b63b17e7d46cc40b0aa24/74835e73-7b6b-41ca-a22b-81c38ed63557.png align="center")
```bash
gcloud artifacts repositories create docker-repo \
--repository-format=docker \
--location=asia-south1
```
Configure Docker:
```bash
gcloud auth configure-docker asia-south1-docker.pkg.dev
```
Build & push:
```bash
docker build -t asia-south1-docker.pkg.dev/gitops-self-healing/docker-repo/service-a:v1 .
docker push asia-south1-docker.pkg.dev/gitops-self-healing/docker-repo/service-a:v1
```
### ❗ ERROR 2 — ImagePullBackOff
Cause: GKE nodes don’t have permission to pull images.
Get project number:
```bash
gcloud projects describe gitops-self-healing \
--format="value(projectNumber)"
```
Grant permission:
```bash
gcloud projects add-iam-policy-binding gitops-self-healing \
--member="serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com" \
--role="roles/artifactregistry.reader"
```
Restart pod:
```bash
kubectl delete pod #service-a-595cc8c965-tlmrh
kubectl get pods -w
```
* * *
## PHASE 3 - Kubernetes Self-Healing
Deployment with probes:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: service-a
spec:
replicas: 1
selector:
matchLabels:
app: service-a
template:
metadata:
labels:
app: service-a
spec:
containers:
- name: service-a
image: asia-south1-docker.pkg.dev/gitops-self-healing/docker-repo/service-a:v1
ports:
- containerPort: 8080
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 5
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 5
```
Apply:
```bash
kubectl apply -f deployment.yaml
```
Test:
```bash
kubectl set env deployment/service-a FAIL=true
```
🔥 Kubernetes automatically restarts unhealthy pods.
* * *
## PHASE 4 - Auto Scaling (HPA)
Check metrics: (CPU/Memory)
```bash
kubectl top pods
```
HPA File:
```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: service-a-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: service-a
minReplicas: 1
maxReplicas: 5
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
```
Apply HPA:
```bash
kubectl apply -f hpa.yaml
kubectl get hpa -w
```
HPA scales based on CPU metrics from Metrics Server.
* * *
## PHASE 5 - Push to GitHub
Upload files to the GitHub Repository
```bash
git init
git add .
git commit -m "Initial commit"
git remote add origin https://github.com//gitops-repo.git
git push -u origin main
```
* * *
## PHASE 6 - Install Argo CD (GitOps Engine)
![](https://cloudmate-test.s3.us-east-1.amazonaws.com/uploads/covers/685b63b17e7d46cc40b0aa24/9b2dabe6-ca1c-4b55-808e-b86abd811d13.png align="center")
Create namespace:
```bash
kubectl create namespace argocd
```
Install:
```bash
kubectl apply -n argocd \
-f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
```
Verify:
```bash
kubectl get pods -n argocd
```
How you access Argo CD UI:
**Option 1 (Learning / Local / GKE lab)**
```yaml
# Port-forward
kubectl port-forward svc/argocd-server -n argocd 8080:443
# Access
https://localhost:8080
```
**Option 2 — Change Service Type to LoadBalancer (Direct Public IP)**
Instead of port-forward, expose Argo CD externally.
```yaml
# Check service:
kubectl get svc argocd-server -n argocd
#It is probably:
ClusterIP
# Patch it:
kubectl patch svc argocd-server -n argocd \ -p '{"spec": {"type": "LoadBalancer"}}'
# Now check:
kubectl get svc argocd-server -n argocd
# Access:
https://
```
**Option 3 — Use Ingress (Recommended for Real DevOps Setup)**
Instead of exposing service directly, use an Ingress.
This allows:
* Domain name
* HTTPS with TLS
* Multiple apps behind one Load Balancer
In GKE, Ingress uses: Google Cloud Load Balancing
Example Ingress:
```yaml
apiVersion: networking.k8s.io/v1kind: Ingressmetadata: name: argocd-ingress namespace: argocdspec: rules: - host: argocd.yourdomain.com http: paths: - path: / pathType: Prefix backend: service: name: argocd-server port: number: 443
```
***Get Admin Password***
```bash
kubectl get secret argocd-initial-admin-secret \
-n argocd \
-o jsonpath="{.data.password}" | base64 -d
```
Login with:
* Username: `admin`
* Password: *decoded value*
### GitOps Application Creation
* Sync Policy: Automatic
* Auto-Prune: Enabled
* Self-Heal: Enabled
Argo CD now continuously reconciles cluster state with Git.
```yaml
# STEP 1️⃣ What you see after login (IMPORTANT)
You’ll see:
- Empty dashboard
- No applications yet
## STEP 2️⃣ Create your FIRST GitOps Application
Click: + NEW APP
Fill like this:
- Application Name: `service-a`
- Project: `default`
- Sync Policy: `Automatic`
- ✅ Automatic
- ✅ Auto-Prune
- ✅ Self-Heal
- Repository URL:
https://github.com//gitops-repo
- Revision:
`main`
- Path:
`service-a`
📌 This path must contain:
- `deployment.yaml`
- `service.yaml`
- `hpa.yaml`
### Destination
- Cluster URL:
`https://kubernetes.default.svc`
- Namespace:
`default`
Click: CREATE
## STEP 3️⃣ What happens immediately after clicking CREATE
Behind the scenes:
1. Argo CD pulls Git repo
2. Reads YAML files
3. Compares with live cluster
4. Applies manifests
5. Shows app as **Healthy / Synced**
You’ll see:
- Green boxes
- Pod creation in real time
## STEP 4️⃣ Verify from terminal (important habit)
kubectl get pods
kubectl get svc
kubectl get hpa
## STEP 5️⃣ Prove SELF-HEALING
# Break something manually
kubectl delete pod -l app=service-a
Result:
- Pod deleted
- Deployment recreates pod
- Argo CD remains synced
# Create configuration drift
kubectl scale deployment service-a --replicas=5
Watch Argo CD UI: It will revert replicas back to Git value automatically.
```
![](https://cloudmate-test.s3.us-east-1.amazonaws.com/uploads/covers/685b63b17e7d46cc40b0aa24/3cddc484-dbec-4223-9394-f7285fb4a743.png align="center")
### ❗ ERROR 3 — Application Not Syncing
Common causes:
* Wrong repo path
* Wrong branch
* Missing YAML files
* Incorrect namespace
Fix path and resync.
* * *
## PHASE 7 - Expose Service via LoadBalancer
Update Service YAML:
```yaml
apiVersion: v1
kind: Service
metadata:
name: service-a
namespace: default
spec:
selector:
app: service-a # matches your pod labels
ports:
- protocol: TCP
port: 80 # the port clients use to access
targetPort: 5000 # the port your container listens on
type: LoadBalancer # gives an external IP in GKE
```
Commit & push:
```bash
git add service-a/
git commit -m "Change python app port to 5000"
git push origin main
```
Check external IP:
```bash
kubectl get svc service-a
# Access
http://
```
* * *
## PHASE 8 - Multi-Service IoT Micro-services
Implement three services to understand how they communicate internally and add those to the ArgoCD.
service-a | Device Service |
|---|
service-b | Sensor Service |
service-c | Alert Service |
As stated earlier created application, create 3 applications for 3 services like in below image.
![](https://cloudmate-test.s3.us-east-1.amazonaws.com/uploads/covers/685b63b17e7d46cc40b0aa24/de6622c8-832b-462a-a82f-6040ac3ec2dd.png align="center")
## PHASE 9 - Setup n8n
![](https://cloudmate-test.s3.us-east-1.amazonaws.com/uploads/covers/685b63b17e7d46cc40b0aa24/d0f17c98-62aa-44cf-a4fa-11b380eb2779.png align="center")
**Why Add n8n?**
Kubernetes + Argo CD already handle 80% of recovery.
But they **do not**:
* Send Slack alerts
* Trigger email notifications
* Execute external Git rollback
* Call external APIs
* Run conditional business logic
That’s where **n8n** comes in.
### Why Install n8n Inside Kubernetes?
You could run n8n: **Inside Kubernetes**
We choose Kubernetes because:
* Same cluster access
* Easy internal DNS communication
* Scalable
* Production-style deployment
* Strong DevOps portfolio value
### Step-by-Step — Install n8n in Kubernetes
1️⃣ Create Namespace
```bash
kubectl create namespace automation
```
Verify:
```bash
kubectl get ns
```
2️⃣ Create n8n Deployment
Create file: `n8n-deployment.yaml`
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: n8n
namespace: automation
spec:
replicas: 1
selector:
matchLabels:
app: n8n
template:
metadata:
labels:
app: n8n
spec:
containers:
- name: n8n
image: n8nio/n8n:latest
ports:
- containerPort: 5678
env:
- name: N8N_BASIC_AUTH_ACTIVE
value: "true"
- name: N8N_BASIC_AUTH_USER
value: "admin"
- name: N8N_BASIC_AUTH_PASSWORD
value: "admin123"
- name: N8N_SECURE_COOKIE
value: "false"
# ── Add these four lines ──────────────────────────
- name: N8N_EDITOR_BASE_URL
value: "http://35.210.234.209"
- name: WEBHOOK_URL
value: "http://35.210.234.209/"
- name: N8N_HOST
value: "35.210.234.209"
- name: N8N_PROTOCOL
value: "http"
```
Apply:
```bash
kubectl apply -f n8n-deployment.yaml
```
3️⃣ Create Service (NodePort)
Create file: `n8n-service.yaml`
```yaml
apiVersion: v1
kind: Service
metadata:
name: n8n
namespace: automation
spec:
type: NodePort
selector:
app: n8n
ports:
- port: 5678
targetPort: 5678
```
Apply:
```bash
kubectl apply -f n8n-service.yaml
```
Verify:
```bash
kubectl get pods -n automation
kubectl get svc -n automation
```
Access n8n:
```text
http://:
```
### ❗ ERROR — n8n Not Accessible
### Possible causes:
* Firewall blocking NodePort
* Wrong external IP
* Pod not running
* Service type incorrect
Check:
```bash
kubectl describe pod -n automation
kubectl describe svc n8n -n automation
```
### 🟢 Build n8n Workflow — Service Health Monitor
```text
[Schedule Trigger] every 1 min
↓
[HTTP Request] service-a /live
↓
[HTTP Request] service-b /live
↓
[HTTP Request] service-c /live
↓
[Code Node] evaluate results
↓
[IF Node] has Issues?
↓ ↓
[Slack] [End]
```
**Node 1 - Schedule Trigger**
* Trigger Interval: Minutes
* Every: 1
This runs monitoring every 60 seconds.
**Nodes 2, 3, 4 — HTTP Requests**
Each node:
Method: GET
URLs:
```plaintext
http://service-a.automation.svc.cluster.local/live
http://service-b.automation.svc.cluster.local/live
http://service-c.automation.svc.cluster.local/live
```
Name them exactly:
* Check Service A
* Check Service B
* Check Service C
⚠️ IMPORTANT:
Go to **Settings tab → Enable “Continue on Fail”**
Without this, workflow stops on first failure.
**Node 5 — Code Node**
```javascript
const serviceA = $('Check Service A').first();
const serviceB = $('Check Service B').first();
const serviceC = $('Check Service C').first();
const results = [
{ name: 'service-a', data: serviceA },
{ name: 'service-b', data: serviceB },
{ name: 'service-c', data: serviceC },
];
const issues = [];
for (const svc of results) {
if (svc.data.error !== undefined) {
issues.push({
service: svc.name,
detail: svc.data.error?.message || 'No response'
});
}
}
const lines = issues.map(i => `*${i.service}* — DOWN\nDetail: ${i.detail}`).join('\n\n');
return [{
json: {
hasIssues: issues.length > 0,
message: issues.length > 0 ? `🚨 Service Health Alert\n\n${lines}` : 'All OK'
}
}];
```
⚠️ Node names inside `$('...')` must match EXACTLY.
**Node 6 — IF Node**
Condition:
```plaintext
{{ $json.hasIssues }}
```
Operation: `is true`
True → Slack
False → End
**Node 7 — Setup a Slack Bot**
```yaml
1. Go to: https://api.slack.com/apps
2. Create New App: → From scratch
3. OAuth & Permissions(Add scopes):
* chat:write
* chat:write.public
* channels:read
4. Install to workspace:
5. Copy Bot Token (starts with xoxb-):
In n8n:
Settings → Credentials → Slack API → paste token
In Slack:
/invite @n8n-alerts
Slack Node Configuration:
* Resource: Message
* Operation: Send
* Channel: #alerts
* Message: {{ $json.message }}
Activate workflow.
```
**Testing the Workflow**
```yaml
Change:
http://service-a/live
To:
http://service-x/live
```
Execute workflow.
✅ Slack alert should fire. Revert back.
* * *
## PHASE 10 - Realistic Production Architecture
Now your architecture becomes:
```text
Kubernetes
├── Pod crashes
├── Scaling events
├── Restarts
│
│ (metrics / events)
▼
n8n
├── Evaluate logic
├── Notify Slack
├── Optional Git rollback
│
▼
Git
│
▼
Argo CD
└── Re-sync cluster
```
* * *
## What Makes This Production-Grade?
* Self-healing pods
* GitOps reconciliation
* Auto-scaling
* Internal service communication
* Automated alerting
* Event-driven workflows
* Slack integration
* Extensible automation engine