Repository Link: https://github.com/sachindumalshan/gitops-repo.git

Architecture Overview

Here is the system level architecture.

Google Cloud (GKE)
│
├── Kubernetes Cluster (gitops-cluster)
│   ├── default namespace
│   │   ├──── service-a (Device Service)
│   │   │     ├─ app.py
│   │   │     ├─ deployment.yaml
│   │   │     ├─ hpa.yaml
│   │   │     ├─ Dockerfile
│   │   │     └─ service.yaml
│   │   ├──── service-b (Sensor Service)
│   │   │     ├─ app.py
│   │   │     ├─ deployment.yaml
│   │   │     ├─ hpa.yaml
│   │   │     ├─ Dockerfile
│   │   │     └─ service.yaml
│   │   └──── service-c (Alert Service)
│   │         ├─ app.py
│   │         ├─ deployment.yaml
│   │         ├─ hpa.yaml
│   │         ├─ Dockerfile
│   │         └─ service.yaml
│   │
│   ├── argocd namespace
│   │   ├── argocd-server
│   │   ├── argocd-repo-server
│   │   ├── argocd-application-controller
│   │   └── argocd-dex-server
│   │
│   └── automation namespace
│       └── n8n
│
├── Google Artifact Registry
│
└── GitHub (Source of Truth)

End-to-End Flow

Developer pushes code → GitHub
        ↓
Argo CD detects change
        ↓
Argo CD syncs to GKE
        ↓
Pods deployed / updated
        ↓
Kubernetes handles:
    - Self-healing
    - Auto-scaling (HPA)
        ↓
n8n monitors services
        ↓
Slack alerts if failure

PHASE 1 - Install gcloud CLI & Setup GKE

Update System

sudo apt-get update

Install required packages:

sudo apt-get install apt-transport-https ca-certificates gnupg curl

Import Google Cloud Public Key

• For newer distributions:

curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg

• For older distributions:

curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -

• If unsupported:

curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

Add Repository

• Newer systems:

echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list

• Older systems:

echo "deb https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list

Install gcloud

sudo apt-get update && sudo apt-get install google-cloud-cli

Login & Configure Project

gcloud auth login
gcloud projects list
gcloud config set project gitops-self-healing-7687

Enable APIs:

gcloud services enable \
  container.googleapis.com \
  compute.googleapis.com \
  cloudbuild.googleapis.com \
  artifactregistry.googleapis.com \

🟢 Create GKE Cluster

gcloud container clusters create gitops-cluster \
  --zone asia-south1-a \
  --num-nodes 2 \
  --machine-type e2-medium

Get credentials:

gcloud container clusters get-credentials gitops-cluster \
  --zone asia-south1-a

❗ ERROR 1 — GKE Auth Plugin Missing

If you see authentication errors:

Install plugin:

sudo apt-get install google-cloud-cli-gke-gcloud-auth-plugin

Verify:

which gke-gcloud-auth-plugin

Refresh kubeconfig:

gcloud container clusters get-credentials gitops-cluster \
  --zone asia-south1-a

Test:

kubectl get nodes

PHASE 2 - Build & Push Python Microservice

Sample Flask App: app.py

from flask import Flask
import os

app = Flask(__name__)

@app.route("/health")
def health():
    if os.getenv("FAIL") == "true":
        return "FAIL", 500
    return "OK", 200

@app.route("/")
def home():
    return "Service A Running", 200

app.run(host="0.0.0.0", port=8080)

Dockerfile

FROM python:3.11-slim
WORKDIR /app
RUN pip install flask
COPY app.py .
CMD ["python", "app.py"]

Create Artifact Registry

gcloud artifacts repositories create docker-repo \
  --repository-format=docker \
  --location=asia-south1

Configure Docker:

gcloud auth configure-docker asia-south1-docker.pkg.dev

Build & push:

docker build -t asia-south1-docker.pkg.dev/gitops-self-healing/docker-repo/service-a:v1 .
docker push asia-south1-docker.pkg.dev/gitops-self-healing/docker-repo/service-a:v1

❗ ERROR 2 — ImagePullBackOff

Cause: GKE nodes don’t have permission to pull images.

Get project number:

gcloud projects describe gitops-self-healing \
  --format="value(projectNumber)"

Grant permission:

gcloud projects add-iam-policy-binding gitops-self-healing \
  --member="serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com" \
  --role="roles/artifactregistry.reader"

Restart pod:

kubectl delete pod <pods_name> #service-a-595cc8c965-tlmrh
kubectl get pods -w

PHASE 3 - Kubernetes Self-Healing

Deployment with probes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: service-a
spec:
  replicas: 1
  selector:
    matchLabels:
      app: service-a
  template:
    metadata:
      labels:
        app: service-a
    spec:
      containers:
      - name: service-a
        image: asia-south1-docker.pkg.dev/gitops-self-healing/docker-repo/service-a:v1
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 5

Apply:

kubectl apply -f deployment.yaml

Test:

kubectl set env deployment/service-a FAIL=true

🔥 Kubernetes automatically restarts unhealthy pods.

PHASE 4 - Auto Scaling (HPA)

Check metrics: (CPU/Memory)

kubectl top pods

HPA File:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: service-a-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: service-a
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

Apply HPA:

kubectl apply -f hpa.yaml
kubectl get hpa -w

HPA scales based on CPU metrics from Metrics Server.

PHASE 5 - Push to GitHub

Upload files to the GitHub Repository

git init
git add .
git commit -m "Initial commit"
git remote add origin https://github.com/<your-username>/gitops-repo.git
git push -u origin main

PHASE 6 - Install Argo CD (GitOps Engine)

Create namespace:

kubectl create namespace argocd

Install:

kubectl apply -n argocd \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Verify:

kubectl get pods -n argocd

How you access Argo CD UI:

Option 1 (Learning / Local / GKE lab)

# Port-forward
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Access
https://localhost:8080

Option 2 — Change Service Type to LoadBalancer (Direct Public IP)

Instead of port-forward, expose Argo CD externally.

# Check service:
kubectl get svc argocd-server -n argocd

#It is probably:
ClusterIP

# Patch it:
kubectl patch svc argocd-server -n argocd \ -p '{"spec": {"type": "LoadBalancer"}}'

# Now check:
kubectl get svc argocd-server -n argocd

# Access:
https://<external-ip>

Option 3 — Use Ingress (Recommended for Real DevOps Setup)

Instead of exposing service directly, use an Ingress.

This allows:

• Domain name

• HTTPS with TLS

• Multiple apps behind one Load Balancer

In GKE, Ingress uses: Google Cloud Load Balancing

Example Ingress:

apiVersion: networking.k8s.io/v1kind: Ingressmetadata: name: argocd-ingress namespace: argocdspec: rules: - host: argocd.yourdomain.com http: paths: - path: / pathType: Prefix backend: service: name: argocd-server port: number: 443

Get Admin Password

kubectl get secret argocd-initial-admin-secret \
  -n argocd \
  -o jsonpath="{.data.password}" | base64 -d

Login with:

• Username: admin

• Password: decoded value

GitOps Application Creation

• Sync Policy: Automatic

• Auto-Prune: Enabled

• Self-Heal: Enabled

Argo CD now continuously reconciles cluster state with Git.

# STEP 1️⃣ What you see after login (IMPORTANT)
You’ll see:
- Empty dashboard
- No applications yet

## STEP 2️⃣ Create your FIRST GitOps Application

Click: + NEW APP

Fill like this:
- Application Name: `service-a`
- Project: `default`
- Sync Policy: `Automatic`
    - ✅ Automatic
    - ✅ Auto-Prune        
    - ✅ Self-Heal        
- Repository URL:
  https://github.com/<your-username>/gitops-repo
- Revision:
  `main`
- Path:
  `service-a`

📌 This path must contain:
- `deployment.yaml`    
- `service.yaml`    
- `hpa.yaml`
    
### Destination
- Cluster URL:
  `https://kubernetes.default.svc`
- Namespace:
  `default`

Click: CREATE

## STEP 3️⃣ What happens immediately after clicking CREATE

Behind the scenes:
1.  Argo CD pulls Git repo
2.  Reads YAML files    
3.  Compares with live cluster    
4.  Applies manifests   
5.  Shows app as **Healthy / Synced**
    
You’ll see:
- Green boxes
- Pod creation in real time

## STEP 4️⃣ Verify from terminal (important habit)
kubectl get pods
kubectl get svc
kubectl get hpa

## STEP 5️⃣ Prove SELF-HEALING

# Break something manually
kubectl delete pod -l app=service-a

Result:
- Pod deleted
- Deployment recreates pod
- Argo CD remains synced

# Create configuration drift
kubectl scale deployment service-a --replicas=5

Watch Argo CD UI: It will revert replicas back to Git value automatically.

❗ ERROR 3 — Application Not Syncing

Common causes:

• Wrong repo path

• Wrong branch

• Missing YAML files

• Incorrect namespace

Fix path and resync.

PHASE 7 - Expose Service via LoadBalancer

Update Service YAML:

apiVersion: v1
kind: Service
metadata:
  name: service-a
  namespace: default
spec:
  selector:
    app: service-a   # matches your pod labels
  ports:
    - protocol: TCP
      port: 80       # the port clients use to access
      targetPort: 5000  # the port your container listens on
  type: LoadBalancer   # gives an external IP in GKE

Commit & push:

git add service-a/
git commit -m "Change python app port to 5000"
git push origin main

Check external IP:

kubectl get svc service-a

# Access
http://<EXTERNAL-IP>

PHASE 8 - Multi-Service IoT Micro-services

Implement three services to understand how they communicate internally and add those to the ArgoCD.

As stated earlier created application, create 3 applications for 3 services like in below image.

PHASE 9 - Setup n8n

Why Add n8n?

Kubernetes + Argo CD already handle 80% of recovery.

But they do not:

• Send Slack alerts

• Trigger email notifications

• Execute external Git rollback

• Call external APIs

• Run conditional business logic

That’s where n8n comes in.

Why Install n8n Inside Kubernetes?

You could run n8n: Inside Kubernetes

We choose Kubernetes because:

• Same cluster access

• Easy internal DNS communication

• Scalable

• Production-style deployment

• Strong DevOps portfolio value

Step-by-Step — Install n8n in Kubernetes

1️⃣ Create Namespace

kubectl create namespace automation

Verify:

kubectl get ns

2️⃣ Create n8n Deployment

Create file: n8n-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: n8n
  namespace: automation
spec:
  replicas: 1
  selector:
    matchLabels:
      app: n8n
  template:
    metadata:
      labels:
        app: n8n
    spec:
      containers:
      - name: n8n
        image: n8nio/n8n:latest
        ports:
        - containerPort: 5678
        env:
        - name: N8N_BASIC_AUTH_ACTIVE
          value: "true"
        - name: N8N_BASIC_AUTH_USER
          value: "admin"
        - name: N8N_BASIC_AUTH_PASSWORD
          value: "admin123"
        - name: N8N_SECURE_COOKIE
          value: "false"
        # ── Add these four lines ──────────────────────────
        - name: N8N_EDITOR_BASE_URL
          value: "http://35.210.234.209"
        - name: WEBHOOK_URL
          value: "http://35.210.234.209/"
        - name: N8N_HOST
          value: "35.210.234.209"
        - name: N8N_PROTOCOL
          value: "http"

Apply:

kubectl apply -f n8n-deployment.yaml

3️⃣ Create Service (NodePort)

Create file: n8n-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: n8n
  namespace: automation
spec:
  type: NodePort
  selector:
    app: n8n
  ports:
  - port: 5678
    targetPort: 5678

Apply:

kubectl apply -f n8n-service.yaml

Verify:

kubectl get pods -n automation
kubectl get svc -n automation

Access n8n:

http://<node-ip>:<nodeport>

❗ ERROR — n8n Not Accessible

Possible causes:

• Firewall blocking NodePort

• Wrong external IP

• Pod not running

• Service type incorrect

Check:

kubectl describe pod -n automation
kubectl describe svc n8n -n automation

🟢 Build n8n Workflow — Service Health Monitor

[Schedule Trigger] every 1 min
        ↓
[HTTP Request] service-a /live
        ↓
[HTTP Request] service-b /live
        ↓
[HTTP Request] service-c /live
        ↓
[Code Node] evaluate results
        ↓
[IF Node] has Issues?
       ↓           ↓
   [Slack]       [End]

Node 1 - Schedule Trigger

• Trigger Interval: Minutes

• Every: 1

This runs monitoring every 60 seconds.

Nodes 2, 3, 4 — HTTP Requests

Each node:

Method: GET

URLs:

http://service-a.automation.svc.cluster.local/live
http://service-b.automation.svc.cluster.local/live
http://service-c.automation.svc.cluster.local/live

Name them exactly:

• Check Service A

• Check Service B

• Check Service C

⚠️ IMPORTANT:

Go to Settings tab → Enable “Continue on Fail”

Without this, workflow stops on first failure.

Node 5 — Code Node

const serviceA = $('Check Service A').first();
const serviceB = $('Check Service B').first();
const serviceC = $('Check Service C').first();

const results = [
  { name: 'service-a', data: serviceA },
  { name: 'service-b', data: serviceB },
  { name: 'service-c', data: serviceC },
];

const issues = [];

for (const svc of results) {
  if (svc.data.error !== undefined) {
    issues.push({
      service: svc.name,
      detail: svc.data.error?.message || 'No response'
    });
  }
}

const lines = issues.map(i => `*\({i.service}* — DOWN\nDetail: \){i.detail}`).join('\n\n');

return [{
  json: {
    hasIssues: issues.length > 0,
    message: issues.length > 0 ? `🚨 Service Health Alert\n\n${lines}` : 'All OK'
  }
}];

⚠️ Node names inside $('...') must match EXACTLY.

Node 6 — IF Node

Condition:

{{ $json.hasIssues }}

Operation: is true

True → Slack
False → End

Node 7 — Setup a Slack Bot

1. Go to: https://api.slack.com/apps
2. Create New App: → From scratch
3. OAuth & Permissions(Add scopes):
   * chat:write
   * chat:write.public
   * channels:read
4. Install to workspace:
5. Copy Bot Token (starts with xoxb-):

In n8n:
Settings → Credentials → Slack API → paste token

In Slack:
/invite @n8n-alerts

Slack Node Configuration:
* Resource: Message
* Operation: Send
* Channel: #alerts
* Message: {{ $json.message }}

Activate workflow.

Testing the Workflow

Change:
http://service-a/live

To:
http://service-x/live

Execute workflow.

✅ Slack alert should fire. Revert back.

PHASE 10 - Realistic Production Architecture

Now your architecture becomes:

Kubernetes
 ├── Pod crashes
 ├── Scaling events
 ├── Restarts
 │
 │ (metrics / events)
 ▼
n8n
 ├── Evaluate logic
 ├── Notify Slack
 ├── Optional Git rollback
 │
 ▼
Git
 │
 ▼
Argo CD
 └── Re-sync cluster

What Makes This Production-Grade?

• Self-healing pods

• GitOps reconciliation

• Auto-scaling

• Internal service communication

• Automated alerting

• Event-driven workflows

• Slack integration

• Extensible automation engine

Overview: Three-tier distributed system using separate VMs for application, monitoring, and logging - mimicking production infrastructure.

Why This Architecture:

• Separation of Concerns: Each VM has a dedicated role (app/monitoring/logging)

• Scalability: Easy to scale each tier independently

• Observability Pillars: Covers metrics (Prometheus), logs (ELK), and visualization (Grafana/Kibana)

Setup Flow

Purpose: Build infrastructure from golden image → deploy services → integrate monitoring/logging

Phase 1: VM Foundation
├── Create Golden Image Template (base OS with common tools)
├── Clone VMs (app-vm, monitoring-vm, logging-vm)
├── Fix Hostnames & Machine IDs  ⚠️ [ERROR #1] (prevent duplicate identity)
├── Configure Static IPs (stable addressing for monitoring)
└── Create Users & Permissions (security & access control)

Phase 2: Application VM Setup
├── Install Python & Dependencies (runtime environment)
├── Create Flask Application (web app with metrics/logging)
├── Install Node Exporter (system-level metrics)
└── Install & Configure Filebeat  ⚠️ [ERROR #2, #3] (log shipper)

Phase 3: Monitoring VM Setup
├── Install Prometheus (time-series metrics database)
├── Configure Scrape Targets (collect from app-vm)
├── Install Grafana (visualization & dashboards)
└── Create Dashboards (display metrics)

Phase 4: Logging VM Setup
├── Install Elasticsearch (log storage & search)
├── Install Kibana (log visualization)
├── Configure Data Views (index patterns)
└── Verify Log Ingestion (confirm data flow)

Phase 5: Integration & Testing
├── Test Metrics Collection (Prometheus → Grafana)
├── Test Log Shipping (Filebeat → ELK)
└── Create Comprehensive Dashboards (unified view)

Detailed Setup Steps

PHASE 1: VM Foundation

Goal: Create reusable VM template and properly configure cloned instances with unique identities.

1.1 Create Golden Image Template

Purpose: Single source of truth - install once, clone many times. Ensures consistency across all VMs.

# Install base Ubuntu Server 22.04
sudo apt update && sudo apt upgrade -y
sudo apt install -y qemu-guest-agent curl wget vim htop net-tools
sudo systemctl enable qemu-guest-agent

# Optional: Install Docker
sudo apt install -y docker docker-compose

1.2 Clean VM Before Cloning

Purpose: Remove machine-specific identifiers to prevent conflicts when cloning.

sudo cloud-init clean
sudo truncate -s 0 /etc/machine-id
sudo rm -f /var/lib/dbus/machine-id
sudo poweroff

1.3 Convert to Template in Proxmox UI

• Right-click VM → Convert to Template

• Note: Template becomes read-only - cannot boot directly

1.4 Clone VMs

• Clone from template for: app-vm, monitoring-vm, logging-vm

• Use Full Clone (recommended for independent VMs)

• Result: 3 identical VMs that need unique configuration

⚠️ ERROR #1 ENCOUNTERED HERE

Why This Matters: Cloned VMs have identical hostnames and machine-ids, causing:

• Prometheus to see only 1 node instead of 3

• Systemd service conflicts

• Network confusion

1.5 Fix Hostnames & Machine IDs

Purpose: Give each VM unique identity for proper monitoring and logging.

App VM:

sudo hostnamectl set-hostname app-vm
hostnamectl  # Verify

Monitoring VM:

sudo hostnamectl set-hostname monitoring-vm
hostnamectl  # Verify

Logging VM:

sudo hostnamectl set-hostname logging-vm
hostnamectl  # Verify

1.6 Fix /etc/hosts (Each VM)

Purpose: Ensure hostname resolves correctly locally.

sudo nano /etc/hosts

Change:

127.0.1.1 app-server

To:

127.0.1.1 app-vm  # (or monitoring-vm, logging-vm respectively)

1.7 Regenerate machine-id (CRITICAL)

Purpose: Create unique systemd identifier - required for proper journaling and service management. ⚠️ Do NOT manually create IDs - let systemd generate them.

sudo rm -f /etc/machine-id
sudo rm -f /var/lib/dbus/machine-id
sudo systemd-machine-id-setup
cat /etc/machine-id  # Verify unique ID
sudo reboot

After reboot, verify each VM has different machine-id

1.8 Create Common User (All VMs)

Purpose: Standard non-root user for application management and SSH access.

sudo useradd -m -s /bin/bash devops
sudo passwd devops
sudo usermod -aG sudo devops

# Verify
getent passwd devops
id devops

1.9 Set Root Password (Optional)

Purpose: Enable root access for emergency situations (homelab only - disable in production).

sudo passwd -u root
sudo passwd root

1.10 Configure Static IPs

Purpose: Fixed IPs are essential for monitoring systems - DHCP changes would break scrape targets and log shipping.

Identify Network Interface:

ip a  # Note interface name (e.g., ens18)

Edit Netplan (Each VM):

sudo nano /etc/netplan/00-installer-config.yaml

App VM (192.168.8.50):

network:
  version: 2
  renderer: networkd
  ethernets:
    ens18:
      dhcp4: no
      addresses:
        - 192.168.8.50/24
      gateway4: 192.168.8.1
      nameservers:
        addresses:
          - 8.8.8.8
          - 1.1.1.1

Monitoring VM (192.168.8.60):

addresses:
  - 192.168.8.60/24
# (Everything else same)

Logging VM (192.168.8.70):

addresses:
  - 192.168.8.70/24
# (Everything else same)

Apply Configuration:

sudo netplan apply
ip a  # Verify
ip route  # Verify gateway

Test Connectivity:

ping -c 3 192.168.8.1  # Gateway
ping 192.168.8.60      # Monitoring VM
ping 192.168.8.70      # Logging VM

Expected: All pings successful = network ready

1.11 Update /etc/hosts (All VMs)

Purpose: Enable hostname-based communication between VMs (easier than remembering IPs).

sudo nano /etc/hosts

Add:

192.168.8.50 app-vm
192.168.8.60 monitoring-vm
192.168.8.70 logging-vm

Test: ping monitoring-vm should work from any VM

PHASE 2: Application VM Setup

Goal: Deploy Flask web application with Prometheus metrics export and JSON logging.

2.1 Install Python & Dependencies

Purpose: Python runtime for Flask application.

ssh devops@app-vm
sudo apt update
sudo apt install -y python3 python3-pip python3-venv

2.2 Create Application Directory

Purpose: Isolated virtual environment prevents dependency conflicts.

cd ~
mkdir myapp
cd myapp
python3 -m venv venv
source venv/bin/activate

Result: Shell prompt shows (venv) prefix

2.3 Install Python Packages

pip install flask prometheus-client
pip list  # Verify

2.4 Create Flask Application

Purpose: Web app that:

• Serves HTTP requests

• Exposes /metrics for Prometheus

• Writes JSON logs for ELK

nano app.py

Paste the following code:

from flask import Flask, Response, render_template_string
import time
import random
import logging
import json
from datetime import datetime
from prometheus_client import (
    Counter,
    Histogram,
    generate_latest,
    CONTENT_TYPE_LATEST
)

app = Flask(__name__)

# ----------------------
# JSON Logging Setup
# ----------------------
class JsonFormatter(logging.Formatter):
    def format(self, record):
        log_record = {
            "@timestamp": datetime.utcnow().isoformat(),
            "log.level": record.levelname,
            "message": record.getMessage(),
            "logger": record.name
        }
        if hasattr(record, "extra"):
            log_record.update(record.extra)
        return json.dumps(log_record)

handler = logging.FileHandler("/var/log/myapp/app.log")
handler.setFormatter(JsonFormatter())
logger = logging.getLogger("myapp")
logger.setLevel(logging.INFO)
logger.addHandler(handler)
logger.propagate = False

# ----------------------
# Prometheus Metrics
# ----------------------
REQUEST_COUNT = Counter(
    "http_requests_total",
    "Total HTTP requests",
    ["method", "endpoint", "status"]
)

REQUEST_LATENCY = Histogram(
    "http_request_latency_seconds",
    "HTTP request latency in seconds",
    ["endpoint"]
)

# ----------------------
# HTML Template
# ----------------------
HTML_TEMPLATE = """
<!DOCTYPE html>
<html>
<head>
    <title>MyApp - Distributed Monitoring Platform</title>
    <style>
        * {
            margin: 0;
            padding: 0;
            box-sizing: border-box;
        }
        html, body {
            height: 100%;
            overflow: hidden;
            font-family: 'Segoe UI', Arial, sans-serif;
        }
        body {
            background: linear-gradient(135deg, #0d47a1 0%, #1976d2 50%, #42a5f5 100%);
            display: flex;
            align-items: center;
            justify-content: center;
        }
        .container {
            width: 95vw;
            height: 95vh;
            background: rgba(227, 242, 253, 0.95);
            border-radius: 15px;
            padding: 2vh 2vw;
            box-shadow: 0 15px 50px rgba(0, 0, 0, 0.3);
            display: flex;
            flex-direction: column;
        }
        header {
            text-align: center;
            padding-bottom: 1.5vh;
            border-bottom: 3px solid #1976d2;
            margin-bottom: 2vh;
        }
        h1 {
            font-size: 2.5vw;
            color: #0d47a1;
            margin-bottom: 0.5vh;
        }
        .tagline {
            font-size: 1.2vw;
            color: #1565c0;
        }
        .main-content {
            flex: 1;
            display: grid;
            grid-template-columns: 1fr 1fr;
            grid-template-rows: auto 1fr;
            gap: 2vh;
            overflow: hidden;
        }
        .section {
            background: #bbdefb;
            padding: 2vh 1.5vw;
            border-radius: 10px;
            border-left: 5px solid #1976d2;
            overflow: auto;
        }
        .section h2 {
            color: #0d47a1;
            font-size: 1.5vw;
            margin-bottom: 1vh;
        }
        .section p, .section li {
            color: #1565c0;
            font-size: 1vw;
            line-height: 1.5;
        }
        .architecture {
            grid-column: 1 / -1;
            background: #90caf9;
        }
        .vm-grid {
            display: grid;
            grid-template-columns: repeat(3, 1fr);
            gap: 1.5vw;
            margin-top: 1vh;
        }
        .vm-box {
            background: #e3f2fd;
            padding: 1.5vh 1vw;
            border-radius: 8px;
            border: 2px solid #1976d2;
            text-align: center;
        }
        .vm-box h3 {
            color: #0d47a1;
            font-size: 1.3vw;
            margin-bottom: 1vh;
        }
        .vm-box p {
            color: #1565c0;
            font-size: 0.9vw;
            margin: 0.5vh 0;
        }
        .vm-icon {
            font-size: 2.5vw;
            margin-bottom: 1vh;
        }
        .api-list {
            list-style: none;
        }
        .api-item {
            background: #e3f2fd;
            padding: 1vh 1vw;
            margin: 0.8vh 0;
            border-radius: 5px;
            border-left: 3px solid #1976d2;
            display: flex;
            justify-content: space-between;
            align-items: center;
        }
        .api-endpoint {
            font-weight: bold;
            color: #0d47a1;
            font-size: 1vw;
        }
        .api-method {
            background: #1976d2;
            color: white;
            padding: 0.3vh 0.8vw;
            border-radius: 3px;
            font-size: 0.8vw;
        }
        .sample-page {
            display: flex;
            flex-direction: column;
            gap: 1vh;
        }
        .sample-card {
            background: #e3f2fd;
            padding: 1vh 1vw;
            border-radius: 5px;
            border-left: 3px solid #1976d2;
        }
        .sample-card h4 {
            color: #0d47a1;
            font-size: 1.1vw;
            margin-bottom: 0.5vh;
        }
        .sample-card p {
            font-size: 0.9vw;
        }
        footer {
            text-align: center;
            padding-top: 1vh;
            border-top: 2px solid #1976d2;
            color: #1565c0;
            font-size: 0.9vw;
            margin-top: 1.5vh;
        }
    </style>
</head>
<body>
    <div class="container">
        <header>
            <h1>MyApp - Distributed Monitoring Platform</h1>
            <p class="tagline">Three-Tier Architecture | Application • Monitoring • Logging</p>
        </header>

        <div class="main-content">
            <div class="section architecture">
                <h2>🏗️ System Architecture</h2>
                <div class="vm-grid">
                    <div class="vm-box">
                        <div class="vm-icon">🖥️</div>
                        <h3>App VM</h3>
                        <p><strong>Role:</strong> Application Server</p>
                        <p><strong>Stack:</strong> Python Flask</p>
                        <p><strong>Port:</strong> 5000</p>
                        <p><strong>Features:</strong> REST APIs, Metrics Export</p>
                    </div>
                    <div class="vm-box">
                        <div class="vm-icon">📊</div>
                        <h3>Monitor VM</h3>
                        <p><strong>Role:</strong> Metrics & Visualization</p>
                        <p><strong>Stack:</strong> Prometheus + Grafana</p>
                        <p><strong>Ports:</strong> 9090, 3000</p>
                        <p><strong>Features:</strong> Time-series DB, Dashboards</p>
                    </div>
                    <div class="vm-box">
                        <div class="vm-icon">📝</div>
                        <h3>Logging VM</h3>
                        <p><strong>Role:</strong> Log Aggregation</p>
                        <p><strong>Stack:</strong> Elasticsearch + Kibana</p>
                        <p><strong>Ports:</strong> 9200, 5601</p>
                        <p><strong>Features:</strong> Log Search, Analysis</p>
                    </div>
                </div>
            </div>

            <div class="section">
                <h2>🔌 Available APIs</h2>
                <ul class="api-list">
                    <li class="api-item">
                        <span class="api-endpoint">/</span>
                        <span class="api-method">GET</span>
                    </li>
                    <li class="api-item">
                        <span class="api-endpoint">/api</span>
                        <span class="api-method">GET</span>
                    </li>
                    <li class="api-item">
                        <span class="api-endpoint">/slow</span>
                        <span class="api-method">GET</span>
                    </li>
                    <li class="api-item">
                        <span class="api-endpoint">/error</span>
                        <span class="api-method">GET</span>
                    </li>
                    <li class="api-item">
                        <span class="api-endpoint">/metrics</span>
                        <span class="api-method">GET</span>
                    </li>
                </ul>
            </div>

            <div class="section sample-page">
                <h2>📄 Sample Page</h2>
                <div class="sample-card">
                    <h4>Application Features</h4>
                    <p>Real-time monitoring with Prometheus metrics collection and Grafana visualization</p>
                </div>
                <div class="sample-card">
                    <h4>Logging System</h4>
                    <p>Centralized log management using Elasticsearch with Kibana dashboards</p>
                </div>
                <div class="sample-card">
                    <h4>Performance Tracking</h4>
                    <p>Request latency, error rates, and throughput metrics tracked across all endpoints</p>
                </div>
                <div class="sample-card">
                    <h4>Distributed Architecture</h4>
                    <p>Scalable three-tier setup with dedicated VMs for app, monitoring, and logging</p>
                </div>
            </div>
        </div>

        <footer>
            <p>🚀 MyApp v1.0 | Powered by Flask • Prometheus • Grafana • Elasticsearch • Kibana | Status: ✅ Running</p>
        </footer>
    </div>
</body>
</html>
"""

# ----------------------
# Routes
# ----------------------
@app.route("/")
def home():
    start_time = time.time()
    REQUEST_COUNT.labels("GET", "/", "200").inc()
    latency = time.time() - start_time
    REQUEST_LATENCY.labels("/").observe(latency)

    logger.info(
        "request_completed",
        extra={
            "endpoint": "/",
            "method": "GET",
            "status": 200,
            "latency_ms": round(latency * 1000, 2)
        }
    )

    return render_template_string(HTML_TEMPLATE)

@app.route("/api")
def api():
    start_time = time.time()
    REQUEST_COUNT.labels("GET", "/api", "200").inc()
    latency = time.time() - start_time
    REQUEST_LATENCY.labels("/api").observe(latency)

    logger.info(
        "request_completed",
        extra={
            "endpoint": "/api",
            "method": "GET",
            "status": 200,
            "latency_ms": round(latency * 1000, 2)
        }
    )

    return "API is running\n"

@app.route("/slow")
def slow():
    delay = random.uniform(1, 4)
    time.sleep(delay)
    REQUEST_COUNT.labels("GET", "/slow", "200").inc()
    REQUEST_LATENCY.labels("/slow").observe(delay)

    logger.warning(
        "slow_request",
        extra={
            "endpoint": "/slow",
            "method": "GET",
            "status": 200,
            "latency_ms": round(delay * 1000, 2)
        }
    )

    return f"Slow response: {delay:.2f}s\n"

@app.route("/error")
def error():
    REQUEST_COUNT.labels("GET", "/error", "500").inc()

    logger.error(
        "application_error",
        extra={
            "endpoint": "/error",
            "method": "GET",
            "status": 500
        }
    )

    return "Error occurred\n", 500

@app.route("/metrics")
def metrics():
    return Response(
        generate_latest(),
        mimetype=CONTENT_TYPE_LATEST
    )

# ----------------------
# Application Entry
# ----------------------
if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

App Features:

• / - Home page with system info

• /api - Simple API endpoint

• /slow - Simulates slow requests (1-4s)

• /error - Returns 500 error

• /metrics - Prometheus metrics endpoint

2.5 Create Log Directory

Purpose: Application needs write permissions for log file.

sudo mkdir -p /var/log/myapp
sudo chown -R devops:devops /var/log/myapp

2.6 Test Application Manually

Purpose: Verify app works before creating systemd service.

python3 app.py

From your laptop:

curl http://192.168.8.50:5000
curl http://192.168.8.50:5000/metrics
cat /var/log/myapp/app.log  # Verify logs

Expected: HTTP responses and JSON logs being written

⚠️ ERROR #3 ENCOUNTERED HERE

2.7 Create Systemd Service

Purpose: Auto-start Flask app on boot and keep it running. Production standard vs manual python app.py.

sudo nano /etc/systemd/system/myapp.service

Paste:

[Unit]
Description=MyApp Flask Application
After=network.target

[Service]
Type=simple
User=devops
Group=devops
WorkingDirectory=/home/devops/myapp
ExecStart=/home/devops/myapp/venv/bin/python3 /home/devops/myapp/app.py

Restart=always
RestartSec=5

StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

Critical Line: ExecStart must point to venv Python, not system Python (see Error #3)

Enable and Start:

sudo systemctl daemon-reload
sudo systemctl enable myapp.service
sudo systemctl start myapp.service
sudo systemctl status myapp.service

Expected: Status shows "active (running)"

2.8 Install Node Exporter

Purpose: Export system metrics (CPU, memory, disk) to Prometheus - app metrics come from Flask, system metrics from Node Exporter.

wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar xvf node_exporter-1.6.1.linux-amd64.tar.gz
sudo mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
sudo useradd --no-create-home --shell /bin/false node_exporter

Create Service:

sudo nano /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Start Service:

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
curl http://localhost:9100/metrics  # Verify

Expected: Hundreds of metrics like node_cpu_seconds_total, node_memory_MemAvailable_bytes

⚠️ ERROR #2 ENCOUNTERED HERE

2.9 Install Filebeat

Purpose: Lightweight log shipper - tails log files and sends to Elasticsearch. Part of the Elastic Stack.

Issue: Filebeat not in standard Ubuntu repos - requires Elastic repository.

# Fix apt repositories first
sudo apt install -y apt-transport-https curl gnupg
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | \
sudo gpg --dearmor -o /usr/share/keyrings/elastic.gpg
echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | \
sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update
sudo apt install filebeat -y

2.10 Configure Filebeat

Purpose: Tell Filebeat what to read (app.log) and where to send (Elasticsearch on logging-vm).

sudo nano /etc/filebeat/filebeat.yml

Minimal Config:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/myapp/app.log
  fields:
    service: my-python-app
  fields_under_root: true

output.elasticsearch:
  hosts: ["http://<ELK_VM_IP>:9200"]

setup.kibana:
  host: "http://<ELK_VM_IP>:5601"

Key Points:

• Input: Monitor /var/log/myapp/app.log

• Output: Send to Elasticsearch at 192.168.8.70:9200

Start Filebeat:

sudo systemctl enable filebeat
sudo systemctl start filebeat
sudo journalctl -u filebeat -f  # Monitor logs

Expected Output:

• "Publishing events"

• "Connection to Elasticsearch established"

• No "connection refused" errors

PHASE 3: Monitoring VM Setup

Goal: Deploy Prometheus (metrics storage) and Grafana (visualization) to monitor app-vm.

3.1 Install Prometheus

Purpose: Time-series database that pulls metrics from app-vm every 15 seconds. Industry standard for metrics.

ssh devops@monitoring-vm
sudo apt update && sudo apt upgrade -y
sudo useradd --no-create-home --shell /bin/false prometheus

Download and Install:

wget https://github.com/prometheus/prometheus/releases/download/v2.47.0/prometheus-2.47.0.linux-amd64.tar.gz
tar xvf prometheus-2.47.0.linux-amd64.tar.gz
sudo mv prometheus-2.47.0.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-2.47.0.linux-amd64/promtool /usr/local/bin/

Create Directories:

sudo mkdir /etc/prometheus
sudo mv prometheus-2.47.0.linux-amd64/consoles /etc/prometheus/
sudo mv prometheus-2.47.0.linux-amd64/console_libraries /etc/prometheus/
sudo mv prometheus-2.47.0.linux-amd64/prometheus.yml /etc/prometheus/
sudo chown -R prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool

3.2 Configure Prometheus

Purpose: Define scrape targets - tell Prometheus where to collect metrics from.

sudo nano /etc/prometheus/prometheus.yml

Add Scrape Targets:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'flask_app'
    static_configs:
      - targets: ['192.168.8.50:5000']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['192.168.8.50:9100']

What This Does:

• Every 15s, scrape 192.168.8.50:5000/metrics (Flask app metrics)

• Every 15s, scrape 192.168.8.50:9100/metrics (system metrics)

• Store data in time-series database

3.3 Create Prometheus Service

sudo nano /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus Monitoring
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file /etc/prometheus/prometheus.yml \
  --storage.tsdb.path /var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Create Storage & Start:

sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus
sudo systemctl status prometheus

Test: Open http://192.168.8.60:9090

• Go to Status → Targets

• Both targets should be "UP"

3.4 Install Grafana

Purpose: Visualization layer on top of Prometheus - creates beautiful dashboards from metrics.

sudo apt update
sudo apt install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
sudo apt update
sudo apt install -y grafana

Start Grafana:

sudo systemctl start grafana-server
sudo systemctl enable grafana-server
sudo systemctl status grafana-server

Access: http://192.168.8.60:3000

• Default login: admin/admin

• You'll be prompted to set new password

3.5 Configure Grafana

1. Add Prometheus Data Source:

• Settings → Data Sources → Add data source

• Select Prometheus

• URL: http://localhost:9090

• Click Save & Test (should show green checkmark)

2. Import Node Exporter Dashboard:

• Dashboards → Import

• Dashboard ID: 1860 (Node Exporter Full)

• Select Prometheus data source

• Import

Result: System metrics dashboard (CPU, memory, disk, network) for app-vm

3. Create Custom Flask Dashboard:

Purpose: Monitor application-specific metrics not covered by Node Exporter.

• Create new dashboard

• Add panel with queries:

Request Rate:

rate(http_requests_total[1m])

Latency (95th percentile):

histogram_quantile(0.95, sum(rate(http_request_latency_seconds_bucket[5m])) by (le))

Error Rate:

rate(http_requests_total{status="500"}[1m])

Why Two Dashboards:

• Dashboard 1860: System health (CPU, RAM, disk)

• Custom dashboard: App health (requests, latency, errors)

PHASE 4: Logging VM Setup

Goal: Deploy ELK stack (Elasticsearch + Kibana) for centralized log management and analysis.

4.1 Install Elasticsearch

Purpose: Scalable search engine - stores and indexes logs for fast querying. Core of the ELK stack.

ssh devops@logging-vm
sudo apt update && sudo apt upgrade -y

# Add Elastic repo
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic.gpg
echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | \
sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update
sudo apt install elasticsearch -y

4.2 Configure Elasticsearch

Purpose: Make Elasticsearch accessible from network and disable security for homelab simplicity.

sudo nano /etc/elasticsearch/elasticsearch.yml

Set:

cluster.name: my-logging-cluster
node.name: logging-node-1

network.host: 0.0.0.0
http.port: 9200

discovery.type: single-node

xpack.security.enabled: false

Configuration Explained:

• network.host: 0.0.0.0 - Accept connections from any IP

• discovery.type: single-node - Not clustering (single VM)

• xpack.security.enabled: false - Disable auth (⚠️ production should enable)

Start Elasticsearch:

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
curl http://localhost:9200  # Verify

Expected: JSON response with cluster name, version info

4.3 Install Kibana

Purpose: Web UI for Elasticsearch - search, visualize, and analyze logs through dashboards.

sudo apt install kibana -y

Configure:

sudo nano /etc/kibana/kibana.yml

server.port: 5601
server.host: "0.0.0.0"

elasticsearch.hosts: ["http://localhost:9200"]

Start Kibana:

sudo systemctl enable kibana
sudo systemctl start kibana
sudo systemctl status kibana

Access: http://192.168.8.70:5601

• Initial load may take 1-2 minutes

4.4 Create Data View in Kibana

Purpose: Tell Kibana which Elasticsearch indices to query. Filebeat creates indices like filebeat-2026.02.07.

1. Go to Stack Management → Data Views

2. Click Create data view

3. Fill in:

   • Name: filebeat-myapp

   • Index pattern: filebeat-* (matches all filebeat indices)

   • Time field: @timestamp

4. Click Save

Why filebeat-* Pattern:

• Filebeat creates daily indices: filebeat-2026.02.07, filebeat-2026.02.08, etc.

• Wildcard * matches all of them

• New indices auto-included

4.5 View Logs

Purpose: Verify logs are flowing from app-vm → Filebeat → Elasticsearch → Kibana.

• Navigate to Discover

• Select data view: filebeat-myapp

• You should see JSON logs with fields:

  • @timestamp - When log was created

  • log.level - INFO, WARNING, ERROR

  • message - Log message

  • endpoint - Which API endpoint

  • latency_ms - Request duration

If No Logs Appear:

• Check Filebeat status on app-vm: sudo systemctl status filebeat

• Check Elasticsearch indices: curl http://192.168.8.70:9200/_cat/indices?v

• Look for filebeat-* indices

PHASE 5: Integration & Testing

Goal: Verify complete data flow and create comprehensive monitoring dashboards.

5.1 Test Complete Flow

Purpose: Generate realistic traffic to produce metrics and logs for visualization.

Generate Traffic:

# From your laptop
for i in {1..100}; do curl http://192.168.8.50:5000/; done
for i in {1..50}; do curl http://192.168.8.50:5000/slow; done
for i in {1..20}; do curl http://192.168.8.50:5000/error; done

What This Creates:

• 100 normal requests → http_requests_total metric increments

• 50 slow requests → http_request_latency_seconds histogram data

• 20 errors → ERROR level logs in Kibana

5.2 Verify Metrics in Grafana

Purpose: Confirm Prometheus is scraping and Grafana is displaying metrics.

• Check http://192.168.8.60:3000

• Verify dashboards show:

  • Request rate: Should spike during traffic generation

  • Latency percentiles: /slow endpoint shows higher latency

  • Error rate: Spike from /error requests

  • System metrics: CPU/memory usage from Node Exporter (Dashboard 1860)

Queries to Verify:

# In Grafana Explore
rate(http_requests_total[1m])           # Should show recent activity
http_request_latency_seconds{quantile="0.95"}  # Should be higher for /slow

5.3 Verify Logs in Kibana

Purpose: Confirm Filebeat → Elasticsearch → Kibana pipeline is working.

• Check http://192.168.8.70:5601

• Go to Discover → Select filebeat-myapp

Search Examples:

log.level: ERROR                    # Find all errors
endpoint: "/slow"                   # Find slow requests
latency_ms > 1000                   # Requests over 1 second

Create Visualizations:

1. Errors Over Time:

   • Lens → Line chart

   • Filter: log.level : "ERROR"

   • X-axis: @timestamp

2. Requests by Endpoint:

   • Lens → Bar chart

   • Y-axis: Count

   • X-axis: endpoint.keyword

3. Latency Distribution:

   • Filter: latency_ms exists

   • Histogram of latency values

Save to Dashboard: Combine visualizations into unified logging dashboard

⚠️ Errors & Solutions Summary

ERROR #1: Duplicate Machine IDs & Hostnames After Cloning

Symptom: All VMs report same hostname and machine-id after cloning from template.

Why This Happens: Proxmox clones EVERYTHING including /etc/machine-id, /etc/hostname, and system identifiers.

Impact:

• Prometheus sees only 1 node instead of 3 (metrics collision)

• Systemd services conflict

• Logs from all VMs appear to come from same source

• Network confusion in monitoring tools

Root Cause: Machine-specific files were copied during clone operation.

Solution:

# Fix hostname
sudo hostnamectl set-hostname <vm-name>

# Fix /etc/hosts
sudo nano /etc/hosts
# Change 127.0.1.1 to correct hostname

# Regenerate machine-id (CRITICAL - must be systemd-generated)
sudo rm -f /etc/machine-id
sudo rm -f /var/lib/dbus/machine-id
sudo systemd-machine-id-setup
sudo reboot

Verification:

# On each VM, these should be DIFFERENT:
hostnamectl
cat /etc/machine-id

Why This is Critical for Observability:

• Prometheus labels nodes by machine-id

• Grafana dashboards group by hostname

• ELK logs tagged with host.name

• Without unique IDs, all data collapses into single source

ERROR #2: Apt Timeout When Installing Filebeat

Symptom:

E: Failed to fetch ...
E: Unable to fetch some archives
Timeout was reached

Why This Happens:

• Using slow/blocked regional mirrors (lk.archive.ubuntu.com)

• Missing Elastic repository (Filebeat not in standard Ubuntu repos)

• Network routing issues for HTTP/HTTPS

Impact: Cannot install Filebeat, blocking log shipping pipeline.

Root Cause: Two-part problem:

1. Ubuntu mirrors unreachable/slow

2. Elastic repo not configured

Solution:

# Step 1: Fix Ubuntu repositories
sudo cp /etc/apt/sources.list /etc/apt/sources.list.bak
sudo nano /etc/apt/sources.list

# Replace with main Ubuntu mirrors:
deb http://archive.ubuntu.com/ubuntu/ jammy main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu jammy-security main restricted universe multiverse

# Step 2: Add Elastic repository
sudo apt install -y apt-transport-https curl gnupg
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | \
sudo gpg --dearmor -o /usr/share/keyrings/elastic.gpg
echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | \
sudo tee /etc/apt/sources.list.d/elastic-8.x.list

# Step 3: Update and install
sudo apt update
sudo apt install filebeat -y

Verification:

filebeat version
# Should show: filebeat version 8.x.x

Why This Matters:

• Filebeat is log shipper - critical component of ELK pipeline

• Without it, logs stay local on app-vm

• No centralized logging = harder debugging in distributed systems

ERROR #3: Flask Service Failed - ModuleNotFoundError

Symptom:

ModuleNotFoundError: No module named 'flask'
systemctl status myapp.service → failed (code=exited, status=1)

Why This Happens: Systemd service points to system Python (/usr/bin/python3) instead of virtual environment Python.

How to Identify:

# Flask installed in venv:
/home/devops/myapp/venv/bin/python3 -c "import flask; print('OK')"
# Returns: OK

# System Python doesn't have Flask:
/usr/bin/python3 -c "import flask"
# Returns: ModuleNotFoundError

Root Cause: Virtual environment isolates dependencies. Systemd service must use venv Python, not system Python.

Incorrect Service File:

ExecStart=/usr/bin/python3 /home/devops/myapp/app.py
# ❌ Uses system Python → no Flask module

Correct Service File:

ExecStart=/home/devops/myapp/venv/bin/python3 /home/devops/myapp/app.py
# ✅ Uses venv Python → Flask available

Full Fix:

sudo systemctl stop myapp.service
sudo nano /etc/systemd/system/myapp.service
# Update ExecStart line to use venv/bin/python3
sudo systemctl daemon-reload
sudo systemctl start myapp.service
sudo systemctl status myapp.service

Verification:

# Service should show "active (running)"
sudo systemctl status myapp.service

# Test endpoint
curl http://localhost:5000
# Should return HTML response

Why This Matters:

• Common mistake when deploying Python apps

• Virtual environments prevent dependency conflicts

• Production best practice: isolate app dependencies

• Systemd service must match development environment

Prevention: Always specify full path to venv Python in systemd services.

🎯 Key Endpoints

This comprehensive guide walks you through building a production-grade cloud-native microservice using FastAPI, Docker, Kubernetes (AWS EKS), and GitHub Actions CI/CD. You'll learn how to implement enterprise-level DevOps practices including automated testing, container orchestration, auto-scaling, health monitoring, and deployment notifications.

What You'll Build:

• Automated CI/CD pipeline using GitHub Actions

• Kubernetes deployment on AWS EKS with auto-scaling

• Multi-environment setup (dev, staging, production)

• Production observability and notifications

Tech Stack: Python, FastAPI, Docker, Kubernetes, AWS EKS, ECR, GitHub Actions, Prometheus, n8n

GitHub Repository: https://github.com/sachindumalshan/cloud-native-microservice-pipeline-monitor

📑 Table of Contents

1. Prerequisites

2. Phase 1: Local Development Setup

3. Phase 2: Containerization with Docker

4. Phase 3: Version Control & CI Setup

5. Phase 4: AWS Infrastructure Setup

6. Phase 5: Kubernetes Deployment

7. Phase 6: Production Reliability Features

8. Phase 7: Observability & Monitoring

9. Phase 8: Multi-Environment Deployment

10. Phase 9: Deployment Notifications

11. Common Errors & Solutions

12. Conclusion & Key Takeaways

Prerequisites

Before starting, ensure you have:

• Python 3.12+ installed

• Docker Desktop running

• AWS Account with appropriate permissions

• GitHub account

• Basic knowledge of Python, REST APIs, and command line

• kubectl and AWS CLI installed

Phase 1: Local Development Setup

Step 1.1: Create Project Structure

mkdir health_metrics_service
cd health_metrics_service
python3 -m venv venv
source venv/bin/activate

Step 1.2: Install Dependencies

pip install fastapi uvicorn pytest

Step 1.3: Create FastAPI Application

Create app.py:

from fastapi import FastAPI
import random, time

app = FastAPI()

@app.get("/health")
def health_check():
    return {"status": "healthy"}

@app.get("/metrics")
def metrics():
    cpu_load = random.uniform(0, 100)
    memory_usage = random.uniform(0, 100)
    return {"cpu": cpu_load, "memory": memory_usage}

@app.post("/simulate_load")
def simulate_load(duration: int = 5):
    start = time.time()
    while time.time() - start < duration:
        sum([i**2 for i in range(10000)])
    return {"status": "load simulated"}

Step 1.4: Test Locally

uvicorn app:app --reload

Visit http://127.0.0.1:8000/health and http://127.0.0.1:8000/metrics to verify.

Step 1.5: Create Unit Tests

Create tests/test_health.py:

from fastapi.testclient import TestClient
from app import app

client = TestClient(app)

def test_health():
    response = client.get("/health")
    assert response.status_code == 200
    assert response.json() == {"status": "healthy"}

Create tests/test_metrics.py:

from fastapi.testclient import TestClient
from app import app

client = TestClient(app)

def test_metrics():
    response = client.get("/metrics")
    assert response.status_code == 200
    data = response.json()
    assert "cpu" in data
    assert "memory" in data

Run tests:

pytest tests/

Step 1.6: Generate Requirements

pip freeze > requirements.txt

Phase 2: Containerization with Docker

Step 2.1: Create Dockerfile

Create Dockerfile:

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

EXPOSE 8000

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

Step 2.2: Build Docker Image

docker build -t health-metrics-service:1.0 .

Step 2.3: Run Container

docker run -d -p 8000:8000 --name health-metrics health-metrics-service:1.0

Step 2.4: Verify Container

docker ps
curl http://localhost:8000/health
docker logs health-metrics

Phase 3: Version Control & CI Setup

Step 3.1: Initialize Git Repository

git init
git add .
git commit -m "Initial commit: FastAPI health & metrics microservice"

Step 3.2: Create GitHub Repository

Create a new repository on GitHub named cloud-native-microservice-pipeline-monitor

git remote add origin https://github.com/<your-username>/cloud-native-microservice-pipeline-monitor.git
git branch -M main
git push -u origin main

Step 3.3: Setup GitHub Actions CI

Create .github/workflows/ci.yml:

name: CI - FastAPI Tests

on:
  push:
    branches: main
  pull_request:
    branches: main

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: |
          pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run unit tests
        run: 
          PYTHONPATH=. pytest

Step 3.4: Add Docker Build to CI

Update .github/workflows/ci.yml:

name: CI - Test & Docker Build

on:
  push:
    branches: main
  pull_request:
    branches: main

jobs:
  test-and-build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: |
          pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run unit tests
        run: 
          PYTHONPATH=. pytest

      - name: Build Docker image
        run: docker build -t health-metrics-service:ci .

Phase 4: AWS Infrastructure Setup

Step 4.1: Create ECR Repository

1. Navigate to AWS Console → ECR → Repositories

2. Click "Create repository"

3. Repository name: health-metrics-service

4. Region: us-east-1 (or your preferred region)

5. Note the repository URI

Step 4.2: Create IAM User for GitHub Actions

1. AWS Console → IAM → Users → Create user

2. Username: github-actions-ecr

3. Access type: Programmatic access

4. Attach policy: AmazonEC2ContainerRegistryPowerUser

5. Save Access Key ID and Secret Access Key

Step 4.3: Configure GitHub Secrets

Go to GitHub repo → Settings → Secrets and variables → Actions

Add these secrets:

• AWS_ACCESS_KEY_ID

• AWS_SECRET_ACCESS_KEY

• AWS_REGION (e.g., us-east-1)

• ECR_REPOSITORY (e.g., health-metrics-service)

• AWS_ACCOUNT_ID

Step 4.4: Update CI/CD to Push to ECR

Update .github/workflows/ci.yml:

name: CI - Test, Build & Push to ECR

on:
  push:
    branches: main
  pull_request:
    branches: main

jobs:
  build-and-push:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run tests
        run: pytest

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}

      - name: Login to Amazon ECR
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build and push Docker image
        env:
          ECR_REGISTRY: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com
          ECR_REPOSITORY: ${{ secrets.ECR_REPOSITORY }}
          IMAGE_TAG: latest
        run: |
          docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
          docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG

Step 4.5: Create EKS Cluster

1. AWS Console → EKS → Clusters → Create cluster

2. Cluster name: health-metrics-cluster

3. Kubernetes version: Latest stable

4. Configure networking (VPC, subnets)

5. Wait ~10 minutes for cluster creation

Step 4.6: Configure kubectl Access

aws eks update-kubeconfig --name health-metrics-cluster --region us-east-1

Step 4.7: Create EKS Access Entry for GitHub Actions

# Create access entry
aws eks create-access-entry \
  --cluster-name health-metrics-cluster \
  --region us-east-1 \
  --principal-arn arn:aws:iam::<AWS_ACCOUNT_ID>:user/github-actions-ecr \
  --type STANDARD

# Grant admin permissions
aws eks associate-access-policy \
  --cluster-name health-metrics-cluster \
  --region us-east-1 \
  --principal-arn arn:aws:iam::<AWS_ACCOUNT_ID>:user/github-actions-ecr \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope type=cluster

Phase 5: Kubernetes Deployment

Step 5.1: Create Kubernetes Manifests

Create k8s/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: health-metrics-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: health-metrics
  template:
    metadata:
      labels:
        app: health-metrics
    spec:
      containers:
      - name: health-metrics-container
        image: <AWS_ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com/health-metrics-service:latest
        ports:
        - containerPort: 8000

Create k8s/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: health-metrics-service
spec:
  type: LoadBalancer
  selector:
    app: health-metrics
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8000

Step 5.2: Add EKS Deployment to CI/CD

Add to .github/workflows/ci.yml:

      - name: Update kubeconfig for EKS
        run: |
          aws eks update-kubeconfig \
            --region ${{ secrets.AWS_REGION }} \
            --name health-metrics-cluster

      - name: Deploy to EKS
        run: kubectl apply -f k8s/

Step 5.3: Add GitHub Secret for EKS

Add EKS_CLUSTER_NAME: health-metrics-cluster to GitHub Secrets

Phase 6: Production Reliability Features

Step 6.1: Add Health Probes

Update k8s/deployment.yaml container spec:

        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 10
          periodSeconds: 10

        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 5
          periodSeconds: 10

Step 6.2: Configure Resource Limits

Add to container spec in k8s/deployment.yaml:

        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"

Step 6.3: Setup Horizontal Pod Autoscaler

Create k8s/hpa.yaml:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: health-metrics-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: health-metrics-deployment
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Apply:

kubectl apply -f k8s/hpa.yaml

Step 6.4: Configure Rolling Updates

Add to k8s/deployment.yaml spec:

  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

Phase 7: Observability & Monitoring

Step 7.1: Implement Prometheus Metrics

Update requirements.txt:

fastapi
uvicorn
prometheus-client
pytest

Update app.py:

from fastapi import FastAPI, Response
from prometheus_client import Counter, Gauge, generate_latest
import random, time

app = FastAPI()

# Prometheus metrics
REQUEST_COUNT = Counter('app_requests_total', 'Total API requests')
CPU_USAGE = Gauge('cpu_usage_percent', 'CPU usage percent')
MEMORY_USAGE = Gauge('memory_usage_percent', 'Memory usage percent')

@app.get("/health")
def health_check():
    return {"status": "healthy"}

@app.get("/metrics")
def metrics_endpoint():
    CPU_USAGE.set(random.uniform(0, 100))
    MEMORY_USAGE.set(random.uniform(0, 100))
    REQUEST_COUNT.inc()
    return Response(generate_latest(), media_type="text/plain")

@app.post("/simulate_load")
def simulate_load(duration: int = 5):
    start = time.time()
    while time.time() - start < duration:
        sum([i**2 for i in range(10000)])
    return {"status": "load simulated"}

Step 7.2: Update Test for Prometheus Format

Update tests/test_metrics.py:

from fastapi.testclient import TestClient
from app import app

client = TestClient(app)

def test_metrics():
    response = client.get("/metrics")
    assert response.status_code == 200
    content = response.text
    assert "cpu_usage_percent" in content
    assert "memory_usage_percent" in content

Phase 8: Multi-Environment Deployment

Step 8.1: Update CI/CD for Multiple Namespaces

Update .github/workflows/ci.yml:

      - name: Set namespace based on branch
        run: |
          if [ "${GITHUB_REF_NAME}" == "development" ]; then
            NAMESPACE="dev"
          elif [ "${GITHUB_REF_NAME}" == "staging" ]; then
            NAMESPACE="staging"
          elif [ "${GITHUB_REF_NAME}" == "main" ]; then
            NAMESPACE="prod"
          else
            echo "Unknown branch, skipping deployment"
            exit 0
          fi
          echo "Namespace=$NAMESPACE" >> $GITHUB_ENV

      - name: Create namespace if missing
        run: |
          kubectl get namespace ${{ env.Namespace }} || kubectl create namespace ${{ env.Namespace }}

      - name: Deploy to EKS
        run: |
          echo "Deploying to ${{ env.Namespace }} namespace..."
          kubectl apply -f k8s/ -n ${{ env.Namespace }}
          kubectl get all -n ${{ env.Namespace }}

Phase 9: Deployment Notifications

Step 9.1: Setup n8n Webhook

1. Create n8n workflow with Webhook node

2. Add Slack/Email notification node

3. Configure message template

4. Note webhook URL

Step 9.2: Add Notification Step to CI/CD

Add to .github/workflows/ci.yml:

      - name: Notify n8n
        if: always()
        run: |
          STATUS="success"
          if [ "${{ job.status }}" != "success" ]; then
            STATUS="failure"
          fi

          curl -X POST https://<n8n-webhook-url>/deployments \
            -H 'Content-Type: application/json' \
            -d '{
                  "service": "health-metrics-service",
                  "namespace": "${{ env.Namespace }}",
                  "status": "'"$STATUS"'"
                }'

Common Errors & Solutions

Error 1: EKS Authentication Failure

Symptoms:

error: You must be logged in to the server (Unauthorized)
couldn't get current server API group list

Root Cause: Missing EKS access entry for GitHub Actions IAM user

Solution:

# Create access entry
aws eks create-access-entry \
  --cluster-name health-metrics-cluster \
  --region us-east-1 \
  --principal-arn arn:aws:iam::<ACCOUNT_ID>:user/github-actions-ecr \
  --type STANDARD

# Grant permissions
aws eks associate-access-policy \
  --cluster-name health-metrics-cluster \
  --region us-east-1 \
  --principal-arn arn:aws:iam::<ACCOUNT_ID>:user/github-actions-ecr \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope type=cluster

Error 2: Stale Kubeconfig

Symptoms:

Unable to connect to the server: dial tcp: lookup <endpoint> on 127.0.0.53:53: no such host

Solution:

rm -f ~/.kube/config
aws eks update-kubeconfig --name health-metrics-cluster --region us-east-1
kubectl get nodes

Error 3: LoadBalancer Pending

Symptoms:

EXTERNAL-IP: <pending>

Root Cause: AWS Load Balancer Controller not installed

Quick Fix - Use NodePort:

# Switch to NodePort
kubectl patch service health-metrics-service -n prod -p '{"spec":{"type":"NodePort"}}'

# Get NodePort
kubectl get service health-metrics-service -n prod

# Open security group
aws ec2 authorize-security-group-ingress \
  --group-id <node-sg> \
  --protocol tcp \
  --port <node-port> \
  --cidr 0.0.0.0/0

Proper Solution - Install AWS Load Balancer Controller:

# Download IAM policy
curl -o iam_policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.7.1/docs/install/iam_policy.json

# Create policy
aws iam create-policy \
  --policy-name AWSLoadBalancerControllerIAMPolicy \
  --policy-document file://iam_policy.json

# Associate OIDC provider
eksctl utils associate-iam-oidc-provider \
  --region us-east-1 \
  --cluster health-metrics-cluster \
  --approve

# Create service account
eksctl create iamserviceaccount \
  --cluster=health-metrics-cluster \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --role-name AmazonEKSLoadBalancerControllerRole \
  --attach-policy-arn=arn:aws:iam::<ACCOUNT_ID>:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve \
  --region=us-east-1

# Install controller via Helm
helm repo add eks https://aws.github.io/eks-charts
helm repo update

VPC_ID=$(aws eks describe-cluster \
  --name health-metrics-cluster \
  --region us-east-1 \
  --query "cluster.resourcesVpcConfig.vpcId" \
  --output text)

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=health-metrics-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set region=us-east-1 \
  --set vpcId=$VPC_ID

Error 4: JSONDecodeError in Tests

Symptoms:

json.decoder.JSONDecodeError: Expecting value: line 1 column 1

Root Cause: Prometheus metrics return text format, not JSON

Solution:

Update tests/test_metrics.py:

def test_metrics():
    response = client.get("/metrics")
    assert response.status_code == 200
    content = response.text
    assert "cpu_usage_percent" in content
    assert "memory_usage_percent" in content

Error 5: OIDC Provider Missing

Symptoms:

Error: no IAM OIDC provider associated with cluster

Solution:

eksctl utils associate-iam-oidc-provider \
  --region us-east-1 \
  --cluster health-metrics-cluster \
  --approve

Error 6: pytest Cannot Find Module (ModuleNotFoundError)

Symptoms:

ModuleNotFoundError: No module named 'app'
ImportError: cannot import name 'app' from 'app'

Root Cause: Python cannot locate your module because the project root isn't in the PYTHONPATH

Solutions:

Option 1: Quick Fix (Temporary)

PYTHONPATH=. pytest tests/

Option 2: Set PYTHONPATH in Virtual Environment (Persistent)

# Add to your virtual environment activation
echo 'export PYTHONPATH="${PYTHONPATH}:$(pwd)"' >> venv/bin/activate

# Reactivate virtual environment
deactivate
source venv/bin/activate

# Now pytest works normally
pytest tests/

Option 3: Create pytest Configuration File (Recommended)

Create pytest.ini in project root:

[pytest]
pythonpath = .
testpaths = tests

Or create pyproject.toml:

[tool.pytest.ini_options]
pythonpath = ["."]
testpaths = ["tests"]

Conclusion

Developer Push to GitHub
         ↓
GitHub Actions CI/CD
         ↓
Run Tests (pytest)
         ↓
Build Docker Image
         ↓
Push to AWS ECR
         ↓
Deploy to EKS (kubectl)
         ↓
Kubernetes Cluster
  ├─ Deployment (2-5 replicas)
  ├─ HPA (auto-scaling)
  ├─ Service (LoadBalancer)
  └─ Health Probes
         ↓
n8n Notification (Slack/Email)

Resources

• FastAPI Documentation

• Kubernetes Documentation

• AWS EKS Best Practices

• GitHub Actions Documentation

• Prometheus Documentation

1. What is LocalStack?

LocalStack is a fully functional local cloud stack that emulates many AWS services like S3, Lambda, DynamoDB, SQS, SNS, and more.

It helps developers:

• Develop and test AWS applications offline at no cost

• Avoid accidental changes to production AWS environments

You can use LocalStack with tools like the AWS CLI, SDKs, and Terraform just like real AWS!

2. Getting a License

LocalStack has both a Free (Community) version and a Pro version with extra features.

Student tip:

• With the GitHub Student Developer Pack, you can get a free LocalStack Pro license key

• Otherwise, sign up for a 14-day free trial at LocalStack website

After signing up, you'll receive a license key to activate LocalStack Pro.

3. Setting Up LocalStack on Your PC

There are two main ways to install and run LocalStack:

Option 1: Install Directly on Your Machine

# 1. Install Python (3.8 or higher) and pip
# 2. Install LocalStack using pip:
pip install localstack

# 3. Set your token (if you have one):
localstack auth set-token <token>

# 4. Start LocalStack:
localstack start

Option 2: Run Using Docker (Recommended)

# 1. Make sure Docker is installed and running
# 2. Pull the LocalStack Docker image:
docker pull localstack/localstack

# 3. Run LocalStack in a container:
docker run -d -p 4566:4566 -p 4571:4571 localstack/localstack

LocalStack will start running on port 4566 (the main gateway for AWS service emulation).

4. Using the LocalStack Dashboard

Once LocalStack is running, open your browser and go to:

👉 http://localhost:4566

You'll see the LocalStack Web Dashboard, which shows:

• Running status of your LocalStack instance

• List of available AWS services (S3, DynamoDB, etc.)

• Logs and configuration options

And also activate your license key from the dashboard.

5. Accessing LocalStack Using AWS CLI

LocalStack works seamlessly with the AWS CLI.

Step 1: Configure AWS CLI (use dummy credentials):

aws configure

# AWS Access Key ID [None]: test
# AWS Secret Access Key [None]: test
# Default region name [us-east-1]: us-east-1
# Default output format [None]: JSON

Enter any values for Access Key, Secret Key, and Region (e.g., us-east-1).

Step 2: Point to LocalStack

Always add --endpoint-url=http://localhost:4566 to your AWS commands.

Example:

aws --endpoint-url=http://localhost:4566 s3 ls

📦 Example: Create an S3 Bucket in LocalStack

Let's create an S3 bucket locally step by step.

Step 1: Make Sure LocalStack is Running

Verify LocalStack is up (using Docker or direct installation).

Step 2: Create a Bucket

aws --endpoint-url=http://localhost:4566 s3 mb s3://my-local-bucket

Step 3: Verify Bucket Creation

aws --endpoint-url=http://localhost:4566 s3 ls

# 2025-11-13 22:38:01 my-local-bucket

You should see your new bucket listed!

Step 4: Upload a File

echo "Hello LocalStack!" > test.txt
aws --endpoint-url=http://localhost:4566 s3 cp test.txt s3://my-local-bucket/

Step 5: List Files in the Bucket

aws --endpoint-url=http://localhost:4566 s3 ls s3://my-local-bucket/

# 2025-11-13 22:38:52         18 test.txt

✅ Conclusion

LocalStack is a fantastic tool for developers who want to:

• Experiment with AWS locally

• Avoid unnecessary cloud costs

• Test automation pipelines or serverless applications safely

When learning AWS, building a local test environment, or developing CI/CD workflows, LocalStack makes local cloud development easy and efficient.

Use this as the foundation. For the application deployment (frontend, backend, DB schema, monitoring), continue in Article 2 in soon.

Table of contents

• Install Proxmox on your homelab PC

• Access the Proxmox dashboard from another PC

• Proxmox components at a glance

• Overview and architecture (basic flow)

• Create LXC container (ct-db)

• Create an AWS EC2 instance

• Tailscale setup (Proxmox host + EC2)

• Enable forwarding and NAT on Proxmox

• Verify EC2-to-container connectivity

• Troubleshooting: Network connectivity & DB integration

Install Proxmox on your homelab PC

1. Download Proxmox VE ISO

   • https://www.proxmox.com/en/downloads

2. Create a bootable USB from the ISO (Rufus, balenaEtcher, or dd)

3. Boot your homelab PC from USB and follow the Proxmox installer

4. During network setup:

   • Set a static IP in your LAN (e.g., 192.168.8.100)

   • Gateway = your router (e.g., 192.168.8.1)

   • DNS = your router or 8.8.8.8

5. Finish install and reboot. On the host console you’ll see something like:

   • Management URL: https://<proxmox-ip>:8006/

   • Login with the credentials you created

Tip: The first login may show a subscription warning; you can still proceed without a subscription.

Access the Proxmox dashboard from another PC

From another PC in the same network:

• Open a browser and go to: https://<proxmox_ip>:8006/

• Accept the self-signed certificate warning

• Login as the user created during install

Now have access to the Proxmox web UI to create containers and manage networking, storage, and backups.

Proxmox components at a glance

• Node (proxmox): the physical host that runs everything (VMs/containers)

• LXC containers (CTs): lightweight OS environments sharing the host kernel; fast and efficient for services like DBs and monitoring

• SDN: optional virtual networking features (VLANs, overlays); can be ignored initially

• Storage:

  • local (/var/lib/vz): ISOs, templates, backups

  • local-lvm: VM/CT disks (faster for root disks)

Conceptual layout:

Datacenter
└── Node: proxmox (your physical server)
      ├── Containers:
      │     ├── 101 (ct-db)
      ├── Networks:
      │     └── vmbr0 (LAN bridge)
      └── Storages:
              ├── local (ISO/backups)
              └── local-lvm (VM/CT disks)

Overview and architecture (basic flow)

Minimal hybrid layout: one EC2 instance in AWS talks privately (via Tailscale) to a single LXC container (ct-db) on your Proxmox host.

   🌩️ AWS Cloud
┌─────────────────┐
│   EC2 Instance  │  (App / Test Client)
└─────────┬───────┘
          │
    (Tailscale VPN)
          │
┌─────────┴────────-┐
│   Proxmox Host    │
│  ┌─────────────┐  │
│  │ LXC: ct-db  │  │  (MariaDB / PostgreSQL)
│  └─────────────┘  │
└───────────────────┘

Goal: Secure, low-latency private connectivity between cloud and homelab for development and experimentation.

Create LXC container (ct-db)

Create a single lightweight LXC container for the database.

Suggested specs:

• ct-db: 2 vCPU, 2–3 GB RAM, Disk 10 GB, static IP (e.g., 192.168.8.101/24), gateway 192.168.8.1

Steps:

1. Datacenter → proxmox → local (storage) → Templates → download Ubuntu template

2. Create CT → set Hostname ct-db, assign static IP 192.168.8.101/24, gateway 192.168.8.1, bridge vmbr0

3. Resources: 2 cores, 2048 MB RAM, optional 512 MB swap

4. Finish → Start container → Console → verify network: ping 8.8.8.8 -c 5

5. Enable “Start at boot” (ct-db → Options → Start at boot → Enable)

Create an AWS EC2 instance

1. Create a minimal EC2 instance (for testing)

   • AMI: Amazon Linux 2023 (or Ubuntu 22.04 LTS)

   • Type: t2.nano or t2.micro

   • Network: place in your VPC (public or private subnet is fine for testing)

   • Security Group: allow SSH from your IP (port 22)

2. Connect via SSH using your key pair

Tailscale setup (Proxmox host + EC2)

Install and authenticate Tailscale on both the Proxmox host and the EC2 instance using the SAME account.

On Proxmox host (advertise the LXC subnet):

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --advertise-routes=192.168.8.0/24

On EC2 (accept advertised routes):

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --accept-routes

Validation:

tailscale status          # both peers listed
ip route get 192.168.8.101  # should show dev tailscale0

If the route does not appear, re-run EC2 command with --accept-routes or check ACLs in the Tailscale admin console.

Enable forwarding and NAT on Proxmox

By default, Tailscale traffic terminates at the Proxmox host. Enable Linux forwarding and NAT so traffic can reach the LXC bridge (vmbr0) and flow back.

On Proxmox host:

1. Enable forwarding (persistent)

• sysctl: net.ipv4.ip_forward=1

2. NAT and forward rules (iptables examples)

• NAT: iptables -t nat -A POSTROUTING -s 100.64.0.0/10 -o vmbr0 -j MASQUERADE

• Forward allow: iptables -I FORWARD -s 100.64.0.0/10 -d 192.168.8.0/24 -j ACCEPT

• Forward return: iptables -I FORWARD -s 192.168.8.0/24 -d 100.64.0.0/10 -m state --state RELATED,ESTABLISHED -j ACCEPT

Note: 100.64.0.0/10 is the Tailscale CGNAT range. Adjust if you use a different mesh range.

Verify EC2-to-container connectivity

Run these quick checks after enabling routes and NAT:

From EC2 (after Tailscale + NAT):

ping 192.168.8.101

From Proxmox host:

tcpdump -i vmbr0 host 192.168.8.101

Troubleshooting: Network connectivity & DB integration

All connectivity issues from “Network Connectivity & Database Integration Troubleshooting” have been consolidated here.

Symptoms and quick diagnoses

• EC2 → Proxmox host: ping works, but EC2 → ct-db: ping fails

  • Likely cause: EC2 not accepting Tailscale subnet routes; or Proxmox not forwarding/NATing

• App timeout on DB connection

  • Likely causes: wrong host IP, DB bound to localhost, missing user/privileges, route not via tailscale0

Fix 1: Accept subnet routes on EC2

• sudo tailscale up --accept-routes

• Confirm: tailscale status shows accepted routes; ip route get 192.168.8.101 returns dev tailscale0 (table 52)

Fix 2: Enable IP forwarding and NAT on Proxmox

• sysctl: net.ipv4.ip_forward=1 (persist via /etc/sysctl.conf)

• iptables NAT: -t nat -A POSTROUTING -s 100.64.0.0/10 -d 192.168.8.0/24 -j MASQUERADE

• iptables FORWARD: allow 100.64.0.0/10 ↔ 192.168.8.0/24 as shown above

Verification commands

EC2:

ping 192.168.8.101
ip route get 192.168.8.101

Proxmox host:

tailscale status
iptables -L -v -n
iptables -t nat -L -v -n

Final checklist

✅ Proxmox host advertises 192.168.8.0/24 via Tailscale

✅ EC2 accepts routes (ip route shows tailscale0 for 192.168.8.101)

✅ IP forwarding enabled; NAT + FORWARD rules applied

✅ ct-db reachable from EC2 (ping, optional DB port test)

✅ (Optional) MariaDB/PostgreSQL installed and listening on container interface

Next: deploy the frontend/backend, database schema, and monitoring in Article 2.

Table of contents

• Overview and architecture

• Create AWS VPC (public + private subnets)

• Provision EC2 instances (frontend public, backend private)

• Configure frontend → backend communication (Nginx reverse proxy)

• Create two LXC in Proxmox (ct-db, ct-monitor)

• Proxmox host ↔ CT connectivity checks

• Backend EC2 ↔ ct-db over Tailscale

• Backend (Node.js + MariaDB client) setup

• Database on ct-db (MariaDB recommended)

• Monitoring on ct-monitor (Prometheus + Grafana)

• End-to-end test and verification

• Security hardening

Overview and architecture

High-level flow: A public frontend EC2 serves the website and proxies API requests to a private backend EC2. The backend talks to an on‑prem database (ct-db) over a Tailscale mesh to your Proxmox host. Monitoring runs in another LXC (ct-monitor).

Prereqs from Article 1:

• Proxmox with two LXC containers

  • ct-db: 192.168.8.101 (MariaDB/PostgreSQL)

  • ct-monitor: 192.168.8.102 (Prometheus + Grafana)

• Tailscale configured

  • Proxmox advertises 192.168.8.0/24

  • EC2 instances accept routes

• Verified EC2 → ct-db connectivity (ping + TCP/3306)

Create AWS VPC (public + private subnets)

Minimal setup:

1. VPC with one public and one private subnet

   • Public subnet: Internet Gateway + route for 0.0.0.0/0

   • Private subnet: NAT Gateway + route for 0.0.0.0/0 via NAT

2. Route tables

   • Associate the public subnet with the IGW route table

   • Associate the private subnet with the NAT route table

Keep the backend private; you’ll reach it from the frontend using its private IP and an Nginx reverse proxy.

Provision EC2 instances (frontend public, backend private)

Create two EC2 instances in your VPC: one in the public subnet for the frontend and one in the private subnet for the backend.

Frontend EC2 (Public Subnet)

SG-frontend Rules:

• Port 22 (SSH) from your IP

• Port 80 (HTTP) from 0.0.0.0/0

• Port 443 (HTTPS) from 0.0.0.0/0

Backend EC2 (Private Subnet)

SG-backend Rules:

• Port 22 (SSH) from SG-frontend

• Port 3000 (TCP) from SG-frontend

SSH Access and Bastion Setup

Connect to Frontend EC2

Log into the frontend VM using SSH with your PEM key:

ssh -i hybrid-cloud.pem ec2-user@<FRONTEND_PUBLIC_IP>

Connect to Backend EC2 (via bastion)

The backend instance has no public IP, so use the frontend as a bastion host.

Step 1: Copy the PEM key to the frontend instance:

scp -i hybrid-cloud.pem hybrid-cloud.pem ec2-user@<FRONTEND_PUBLIC_IP>:/home/ec2-user/

Step 2: SSH from the frontend to the backend using its private IP:

ssh -i ~/hybrid-cloud.pem ec2-user@<BACKEND_PRIVATE_IP>

Configure Frontend (Nginx + Static Site + Reverse Proxy)

This section covers installing Nginx, creating the static HTML pages, and configuring the reverse proxy to route API requests to the backend.

Install and Enable Nginx

Update packages and install Nginx:

sudo yum update -y && sudo yum upgrade -y
sudo yum install -y nginx
sudo systemctl enable --now nginx

Create Web Root and HTML Files

Create the web root directory:

sudo mkdir -p /var/www/html
cd /var/www/html

Create index.html:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Hybrid Cloud Platform</title>
  <style>
    * {
      margin: 0;
      padding: 0;
      box-sizing: border-box;
    }

    body {
      font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
      background: linear-gradient(135deg, #31694E 0%, #658C58 100%);
      min-height: 100vh;
      display: flex;
      justify-content: center;
      align-items: center;
      padding: 20px;
    }

    .container {
      background: #F0E491;
      border-radius: 20px;
      box-shadow: 0 20px 60px rgba(0, 0, 0, 0.3);
      padding: 60px 40px;
      max-width: 600px;
      width: 100%;
      text-align: center;
    }

    h1 {
      color: #31694E;
      font-size: 2.5em;
      margin-bottom: 15px;
      font-weight: 700;
    }

    .subtitle {
      color: #658C58;
      font-size: 1.2em;
      margin-bottom: 40px;
      font-weight: 500;
    }

    .project-scope {
      background: white;
      border-left: 5px solid #BBC863;
      padding: 20px;
      margin: 30px 0;
      border-radius: 8px;
      text-align: left;
    }

    .project-scope h2 {
      color: #31694E;
      font-size: 1.3em;
      margin-bottom: 10px;
    }

    .project-scope p {
      color: #333;
      line-height: 1.6;
      font-size: 1.05em;
    }

    .tech-badges {
      display: flex;
      justify-content: center;
      gap: 15px;
      margin: 20px 0;
      flex-wrap: wrap;
    }

    .badge {
      background: #658C58;
      color: #F0E491;
      padding: 8px 20px;
      border-radius: 20px;
      font-size: 0.9em;
      font-weight: 600;
    }

    .cta-button {
      display: inline-block;
      background: #31694E;
      color: #F0E491;
      text-decoration: none;
      padding: 18px 50px;
      border-radius: 50px;
      font-size: 1.3em;
      font-weight: 700;
      margin-top: 30px;
      transition: all 0.3s ease;
      box-shadow: 0 8px 20px rgba(49, 105, 78, 0.4);
      position: relative;
      overflow: hidden;
    }

    .cta-button::before {
      content: '';
      position: absolute;
      top: 0;
      left: -100%;
      width: 100%;
      height: 100%;
      background: linear-gradient(90deg, transparent, rgba(255, 255, 255, 0.2), transparent);
      transition: left 0.5s;
    }

    .cta-button:hover::before {
      left: 100%;
    }

    .cta-button:hover {
      background: #254d3a;
      transform: translateY(-3px);
      box-shadow: 0 12px 30px rgba(49, 105, 78, 0.5);
    }

    .highlight-box {
      background: #BBC863;
      color: #31694E;
      padding: 5px 15px;
      border-radius: 5px;
      display: inline-block;
      font-weight: 700;
      margin-top: 10px;
      font-size: 0.95em;
    }

    @media (max-width: 600px) {
      .container {
        padding: 40px 25px;
      }

      h1 {
        font-size: 2em;
      }

      .subtitle {
        font-size: 1em;
      }
    }
  </style>
</head>
<body>
  <div class="container">
    <h1>🚀 Hybrid Cloud Platform</h1>
    <p class="subtitle">Seamless Integration Across Environments</p>

    <div class="project-scope">
      <h2>Project Scope</h2>
      <p><strong>Hybrid Cloud Infrastructure:</strong> AWS Cloud + Proxmox Homelab</p>
      <div class="tech-badges">
        <span class="badge">AWS</span>
        <span class="badge">Proxmox</span>
        <span class="badge">Homelab</span>
      </div>
    </div>

    <a href="register.html" class="cta-button">Register as a User</a>
  </div>
</body>
</html>

Create register.html:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Register - Hybrid Cloud Platform</title>
  <style>
    * {
      margin: 0;
      padding: 0;
      box-sizing: border-box;
    }

    body {
      font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
      background: linear-gradient(135deg, #31694E 0%, #658C58 100%);
      min-height: 100vh;
      display: flex;
      justify-content: center;
      align-items: center;
      padding: 20px;
    }

    .container {
      background: #F0E491;
      border-radius: 20px;
      box-shadow: 0 20px 60px rgba(0, 0, 0, 0.3);
      padding: 50px 40px;
      max-width: 500px;
      width: 100%;
    }

    .header {
      text-align: center;
      margin-bottom: 40px;
    }

    h2 {
      color: #31694E;
      font-size: 2.2em;
      margin-bottom: 10px;
      font-weight: 700;
    }

    .subtitle {
      color: #658C58;
      font-size: 1em;
      margin-bottom: 10px;
    }

    .back-link {
      display: inline-block;
      color: #31694E;
      text-decoration: none;
      font-size: 0.95em;
      margin-bottom: 20px;
      transition: color 0.3s;
    }

    .back-link:hover {
      color: #658C58;
    }

    .form-group {
      margin-bottom: 25px;
    }

    label {
      display: block;
      color: #31694E;
      font-weight: 600;
      margin-bottom: 8px;
      font-size: 0.95em;
    }

    input[type="text"],
    input[type="password"] {
      width: 100%;
      padding: 15px;
      border: 2px solid #BBC863;
      border-radius: 10px;
      font-size: 1em;
      background: white;
      color: #31694E;
      transition: all 0.3s;
      outline: none;
    }

    input[type="text"]:focus,
    input[type="password"]:focus {
      border-color: #658C58;
      box-shadow: 0 0 0 3px rgba(101, 140, 88, 0.1);
    }

    input::placeholder {
      color: #BBC863;
    }

    .btn-submit {
      width: 100%;
      padding: 16px;
      background: #31694E;
      color: #F0E491;
      border: none;
      border-radius: 10px;
      font-size: 1.1em;
      font-weight: 700;
      cursor: pointer;
      transition: all 0.3s;
      box-shadow: 0 6px 20px rgba(49, 105, 78, 0.3);
      margin-top: 10px;
    }

    .btn-submit:hover {
      background: #254d3a;
      transform: translateY(-2px);
      box-shadow: 0 8px 25px rgba(49, 105, 78, 0.4);
    }

    .btn-submit:active {
      transform: translateY(0);
    }

    .btn-submit:disabled {
      background: #BBC863;
      cursor: not-allowed;
      transform: none;
    }

    .info-box {
      background: white;
      border-left: 4px solid #BBC863;
      padding: 15px;
      border-radius: 8px;
      margin-top: 25px;
      font-size: 0.9em;
      color: #31694E;
    }

    .alert {
      padding: 15px;
      border-radius: 8px;
      margin-top: 20px;
      font-weight: 500;
      display: none;
    }

    .alert.success {
      background: #BBC863;
      color: #31694E;
      display: block;
    }

    .alert.error {
      background: #ff6b6b;
      color: white;
      display: block;
    }

    @media (max-width: 600px) {
      .container {
        padding: 35px 25px;
      }

      h2 {
        font-size: 1.8em;
      }
    }
  </style>
</head>
<body>
  <div class="container">
    <a href="index.html" class="back-link">← Back to Home</a>

    <div class="header">
      <h2>Create Account</h2>
      <p class="subtitle">Join the Hybrid Cloud Platform</p>
    </div>

    <form id="regForm">
      <div class="form-group">
        <label for="username">Username</label>
        <input type="text" id="username" placeholder="Enter your username" required>
      </div>

      <div class="form-group">
        <label for="password">Password</label>
        <input type="password" id="password" placeholder="Enter your password" required>
      </div>

      <button type="submit" class="btn-submit">Register Account</button>

      <div id="alertBox" class="alert"></div>
    </form>

    <div class="info-box">
      <strong>🔒 Secure Registration</strong><br>
      Your credentials will be securely stored in our hybrid cloud infrastructure.
    </div>
  </div>

  <script>
    document.getElementById('regForm').addEventListener('submit', async (e) => {
      e.preventDefault();

      const alertBox = document.getElementById('alertBox');
      const submitBtn = e.target.querySelector('.btn-submit');
      const username = document.getElementById('username').value;
      const password = document.getElementById('password').value;

      // Reset alert
      alertBox.className = 'alert';
      alertBox.textContent = '';

      // Disable button during request
      submitBtn.disabled = true;
      submitBtn.textContent = 'Registering...';

      try {
        const res = await fetch('/api/register', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify({ username, password })
        });

        // Check if response is ok
        if (!res.ok) {
          throw new Error(`Server returned ${res.status}: ${res.statusText}`);
        }

        // Check if response is JSON
        const contentType = res.headers.get('content-type');
        if (!contentType || !contentType.includes('application/json')) {
          const text = await res.text();
          throw new Error(`Expected JSON but got: ${text.substring(0, 100)}`);
        }

        const data = await res.json();

        if (data.success) {
          alertBox.className = 'alert success';
          alertBox.textContent = `✓ User ${data.user.username} registered successfully!`;
          document.getElementById('regForm').reset();
        } else {
          alertBox.className = 'alert error';
          alertBox.textContent = `✗ Error: ${data.message}`;
        }
      } catch (error) {
        alertBox.className = 'alert error';
        alertBox.textContent = `✗ Error: ${error.message}`;
        console.error('Registration error:', error);
      } finally {
        // Re-enable button
        submitBtn.disabled = false;
        submitBtn.textContent = 'Register Account';
      }
    });
  </script>
</body>
</html>

Configure Nginx Reverse Proxy

Edit the Nginx configuration (Amazon Linux path: /etc/nginx/nginx.conf or create a file under /etc/nginx/conf.d/):

server {
    listen 80;
    server_name _;

    root /var/www/html;
    index index.html;

    location /api/ {
        proxy_pass http://<BACKEND_PRIVATE_IP>:3000/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Replace <BACKEND_PRIVATE_IP> with your backend EC2's private IP address.

Restart Nginx

Restart Nginx to apply the configuration:

sudo systemctl restart nginx

Test Connectivity

Optional: verify the frontend can reach the backend:

ping <BACKEND_PRIVATE_IP> -c 5

Configure Backend (Node.js API)

This section covers setting up the Node.js backend API that connects to the ct-db database over Tailscale.

Install Node.js and Dependencies

On the backend EC2 instance, install Node.js and create the application directory

sudo yum update -y && sudo yum upgrade -y
sudo yum install -y nodejs npm
mkdir ~/simple-app && cd ~/simple-app
npm init -y
npm install express body-parser cors mariadb
sudo yum install mariadb105 -y

Create the Backend API (server.js)

Create server.js in the ~/simple-app directory:

const express = require('express');
const bodyParser = require('body-parser');
const mariadb = require('mariadb');
const cors = require('cors');
const path = require('path');

const app = express();

// Middleware
app.use(cors());
app.use(bodyParser.json());
app.use(express.static(path.join(__dirname, 'public'))); // Serve static files

// Create MariaDB pool with increased timeouts for VPN connection
const pool = mariadb.createPool({
  host: '192.168.8.101',
  user: 'appuser',
  password: 'StrongPassword123',
  database: 'appdb',
  port: 3306,
  connectionLimit: 10,
  connectTimeout: 30000,        // 30 seconds (default is 1000ms)
  acquireTimeout: 30000,         // 30 seconds for pool acquisition
  timeout: 30000,                // 30 seconds for queries
  socketTimeout: 30000,          // 30 seconds for socket operations
  initializationTimeout: 30000   // 30 seconds for pool initialization
});

// Test database connection on startup
(async () => {
  try {
    const conn = await pool.getConnection();
    console.log('✓ Database connected successfully');
    conn.release();
  } catch (err) {
    console.error('✗ Database connection failed:', err);
  }
})();

// API Routes
app.post('/api/register', async (req, res) => {
  const { username, password } = req.body;

  // Validation
  if (!username || !password) {
    return res.status(400).json({ 
      success: false, 
      message: 'Username and password required' 
    });
  }

  if (username.length < 3) {
    return res.status(400).json({ 
      success: false, 
      message: 'Username must be at least 3 characters' 
    });
  }

  if (password.length < 6) {
    return res.status(400).json({ 
      success: false, 
      message: 'Password must be at least 6 characters' 
    });
  }

  let conn;
  try {
    conn = await pool.getConnection();

    const result = await conn.query(
      'INSERT INTO users (username, password) VALUES (?, ?)',
      [username, password]
    );

    console.log(`✓ User registered: ${username} (ID: ${result.insertId})`);

    res.json({
      success: true,
      message: 'User registered successfully!',
      user: {
        username: username,
        id: result.insertId.toString()
      }
    });

  } catch (err) {
    console.error('Database error:', err);

    // Handle duplicate username
    if (err.code === 'ER_DUP_ENTRY') {
      return res.status(409).json({
        success: false,
        message: 'Username already exists'
      });
    }

    // Generic server error
    res.status(500).json({ 
      success: false, 
      message: 'Server error occurred' 
    });

  } finally {
    if (conn) conn.release();
  }
});

// Get all users (for testing/admin purposes)
app.get('/api/users', async (req, res) => {
  let conn;
  try {
    conn = await pool.getConnection();
    const rows = await conn.query('SELECT id, username, created_at FROM users ORDER BY created_at DESC');

    res.json({
      success: true,
      users: rows
    });

  } catch (err) {
    console.error('Database error:', err);
    res.status(500).json({ 
      success: false, 
      message: 'Server error' 
    });
  } finally {
    if (conn) conn.release();
  }
});

// Health check endpoint
app.get('/api/health', async (req, res) => {
  let dbStatus = 'disconnected';

  try {
    const conn = await pool.getConnection();
    dbStatus = 'connected';
    conn.release();
  } catch (err) {
    console.error('Health check failed:', err);
  }

  res.json({
    status: 'running',
    database: dbStatus,
    timestamp: new Date().toISOString()
  });
});

// 404 handler
app.use((req, res) => {
  res.status(404).json({ 
    success: false, 
    message: 'Endpoint not found' 
  });
});

// Error handler
app.use((err, req, res, next) => {
  console.error('Server error:', err);
  res.status(500).json({ 
    success: false, 
    message: 'Internal server error' 
  });
});

// Start server
const PORT = process.env.PORT || 3000;
app.listen(PORT, '0.0.0.0', () => {
  console.log('=================================');
  console.log(`🚀 Server running on port ${PORT}`);
  console.log(`📡 Access at: http://localhost:${PORT}`);
  console.log('=================================');
});

Run the Backend Server

Start the Node.js server:

node server.js

You should see output like:

=================================
🚀 Server running on port 3000
📡 Access at: http://localhost:3000
=================================

Create Two LXC Containers in Proxmox

This section covers creating two LXC containers on your Proxmox host: one for the database (ct-db) and one for monitoring (ct-monitor).

Download LXC Template

In the Proxmox web UI:

1. Navigate to Datacenter → proxmox → local (storage) → Templates

2. Click Templates button

3. Download a Debian or Ubuntu template (e.g., ubuntu-22.04-standard)

Create ct-db Container (Database)

Click Create CT and configure:

After creation, enable "Start at boot" in ct-db → Options.

Create ct-monitor Container (Monitoring)

Click Create CT and configure:

After creation, enable "Start at boot" in ct-monitor → Options.

Configure Database on ct-db

This section covers installing MariaDB, creating the database and user, and setting up the users table.

Install and Start MariaDB

Access the ct-db container console from Proxmox UI or use:

pct enter 101

Update and install MariaDB:

sudo apt update && sudo apt upgrade -y
sudo apt install -y mariadb-server
sudo systemctl enable mariadb
sudo systemctl start mariadb

Create Database and User

Log into MariaDB:

mysql -u root -p

Run the following SQL commands:

-- Create database
CREATE DATABASE appdb;

-- Show current users and their hosts
SELECT User, Host FROM mysql.user;

-- Create user with remote access
GRANT ALL PRIVILEGES ON appdb.* TO 'appuser'@'%' IDENTIFIED BY 'StrongPassword123';
FLUSH PRIVILEGES;

-- Verify the grant
SELECT User, Host FROM mysql.user WHERE User='appuser';

-- Switch to database
USE appdb;

-- Create users table
CREATE TABLE IF NOT EXISTS users (
  id INT AUTO_INCREMENT PRIMARY KEY,
  username VARCHAR(255) UNIQUE NOT NULL,
  password VARCHAR(255) NOT NULL,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Verify table structure
DESCRIBE users;

EXIT;

Configure Remote Access

Edit the MariaDB configuration to allow remote connections:

sudo nano /etc/mysql/mariadb.conf.d/50-server.cnf

Find the bind-address line and change it to:

bind-address = 0.0.0.0

Restart MariaDB:

sudo systemctl restart mariadb

Configure Monitoring Setup on ct-monitor (Prometheus + Grafana + Node Exporter)

This comprehensive section covers installing and configuring Prometheus, Grafana, and Node Exporter across all machines.

1. Install on ct-monitor (192.168.8.102)

Access the ct-monitor container:

pct enter 102

Install Prometheus

# Access container
pct enter 102

# Update system
apt update && apt upgrade -y

# === PROMETHEUS ===
useradd --no-create-home --shell /bin/false prometheus
cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.47.0/prometheus-2.47.0.linux-amd64.tar.gz
tar -xvf prometheus-2.47.0.linux-amd64.tar.gz
cd prometheus-2.47.0.linux-amd64
mv prometheus promtool /usr/local/bin/
mkdir -p /etc/prometheus /var/lib/prometheus
mv consoles console_libraries /etc/prometheus/
chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

# Create Prometheus config (only 2 targets)
cat > /etc/prometheus/prometheus.yml <<EOF
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'ct-monitor'
    static_configs:
      - targets: ['localhost:9100']

  - job_name: 'backend-ec2'
    static_configs:
      - targets: ['BACKEND_TAILSCALE_IP:9100']
EOF

chown prometheus:prometheus /etc/prometheus/prometheus.yml

# Create Prometheus service
cat > /etc/systemd/system/prometheus.service <<EOF
[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl start prometheus
systemctl enable prometheus

Install Node Exporter

cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar -xvf node_exporter-1.6.1.linux-amd64.tar.gz
mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
useradd --no-create-home --shell /bin/false node_exporter

cat > /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl start node_exporter
systemctl enable node_exporter

Install Grafana

# === GRAFANA ===
apt install -y software-properties-common apt-transport-https wget
wget -q -O - https://packages.grafana.com/gpg.key | apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | tee /etc/apt/sources.list.d/grafana.list
apt update
apt install -y grafana

systemctl start grafana-server
systemctl enable grafana-server

# Verify all services
systemctl status prometheus
systemctl status node_exporter
systemctl status grafana-server

2. Install on Backend EC2

# SSH to Backend EC2
cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar -xvf node_exporter-1.6.1.linux-amd64.tar.gz
sudo mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
sudo useradd --no-create-home --shell /bin/false node_exporter

sudo tee /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
sudo systemctl status node_exporter

# Get Tailscale IP for Prometheus config
tailscale ip -4

3. Update Prometheus Config with Tailscale IPs

# On ct-monitor, get Tailscale IPs from EC2 instances
# Then edit Prometheus config
nano /etc/prometheus/prometheus.yml

# Replace BACKEND_TAILSCALE_IP and FRONTEND_TAILSCALE_IP with actual IPs
# Example:
#   - targets: ['100.117.212.126:9100']  # Backend
#   - targets: ['100.124.182.20:9100']   # Frontend

# Restart Prometheus
systemctl restart prometheus

4. Access Grafana

# Open browser
http://192.168.8.102:3000

# Login: admin / admin
# Change password when prompted

5. Configure Grafana Data Source

1. Login to Grafana

2. Go to Configuration → Data Sources

3. Click Add data source

4. Select Prometheus

5. URL: http://localhost:9090

6. Click Save & Test

6. Import Dashboard

1. Go to Dashboards → Import

2. Enter ID: 1860 (Node Exporter Full)

3. Click Load

4. Select Prometheus data source

5. Click Import

Verification Commands

# Check all services on ct-monitor
systemctl status prometheus
systemctl status node_exporter
systemctl status grafana-server

# Check ports
ss -tulpn | grep -E '9090|9100|3000'

# Test from Backend EC2
curl http://192.168.8.102:9090
curl http://192.168.8.102:3000

Proxmox host ↔ CT connectivity checks

Step 1 — Check container network interface (run on Proxmox host)

pct exec 101 -- ip addr

This shows the container’s eth0 IP and whether the interface is UP. If eth0 is missing or has no IP, it cannot respond.

Step 2 — Check container routing

pct exec 101 -- ip route

You should see something like:

default via 192.168.8.1 dev eth0
192.168.8.0/24 dev eth0 proto kernel scope link src 192.168.8.101

Step 3 — Check bridge connectivity on host

brctl show vmbr0
ip addr show vmbr0

Confirm vmbr0 is UP and the container veth (veth101i0) is attached.

Step 4 — Restart container network stack

Sometimes the network inside a container doesn’t come up automatically after stop/start:

pct exec 101 -- ip link set eth0 up
pct exec 101 -- ip addr add 192.168.8.101/24 dev eth0
pct exec 101 -- ip route add default via 192.168.8.1

Adjust IP/gateway as per your container config.

Now test:

# From Proxmox host
ping 192.168.8.101 -c 3

# Inside container
ping 8.8.8.8 -c 3

Backend EC2 ↔ ct-db over Tailscale

On Proxmox host (advertise the LXC subnet):

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --advertise-routes=192.168.8.0/24

On EC2 (accept advertised routes):

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --accept-routes

On backend EC2, validate routing and DB port reachability:

tailscale status
ip route get 192.168.8.101   # expect dev tailscale0
nc -zv 192.168.8.101 3306    # DB port test (MariaDB/MySQL)

Troubleshooting (see Article 1): ensure IP forwarding + NAT on Proxmox, and --accept-routes on EC2.

Monitoring on ct-monitor (Prometheus + Grafana)

On ct-monitor:

1. Install Prometheus, run as a dedicated user, and expose on 0.0.0.0:9090

2. Configure scrape targets (localhost and any exporters)

3. Install Grafana via apt repo; enable grafana-server

4. Access Grafana from your LAN and add dashboards (system metrics, DB metrics, blackbox checks)

Your notes include working unit files and a minimal prometheus.yml; reuse them here.

End-to-end test and verification

1. From frontend: open the site via public IP; navigate to the registration page

2. Submit a username/password; the request hits frontend /api/register and proxies to backend private IP:3000

3. Backend connects to ct-db at 192.168.8.101:3306 (over Tailscale) and inserts into users

4. Confirm in DB: SELECT * FROM users; on ct-db

5. Optional: Create a Grafana dashboard to visualize backend host metrics

If requests time out or DB connection fails, check Article 1’s troubleshooting section (routing, NAT, bind-address, user grants).

Modern web applications need to handle thousands of concurrent users while maintaining high performance and reliability. This is where nginx shines as one of the most powerful and versatile web servers available today. In this comprehensive guide, we'll explore how nginx functions as a web server, reverse proxy, and load balancer through a hands-on Docker example.

GitHub Repository: https://github.com/sachindumalshan/simple-nginx-app.git

What is nginx?

nginx (pronounced "engine-x") is a high-performance web server, reverse proxy server, and load balancer. Nginx has become one of the most popular web servers in the world, powering over 30% of all websites globally.

Key Features of Nginx:

• High Performance: Can handle thousands of concurrent connections with low memory usage

• Reverse Proxy: Routes client requests to backend servers

• Load Balancing: Distributes incoming requests across multiple servers

• Static Content Serving: Efficiently serves static files like HTML, CSS, and images

• SSL/TLS Termination: Handles encryption and decryption

• Caching: Improves performance by storing frequently requested content

Demo Application Architecture

Before diving into concepts, let's understand a practical example:

┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   Browser   │────│    Nginx    │────│  Backend 1  │
│             │    │  (Port 8080)│    │   (PHP)     │
└─────────────┘    │             │    └─────────────┘
                   │             │    
                   │             │    ┌─────────────┐
                   │             │────│  Backend 2  │
                   └─────────────┘    │   (PHP)     │
                                      └─────────────┘

The application consists of:

• Frontend: Static HTML contact form served by nginx

• Two PHP Back-ends: Process form submissions and show container IDs with a message

• nginx: Acts as web server, reverse proxy, and load balancer

• Docker: Orchestrates all services

Nginx as a Web Server

What is a Web Server?

A web server is software that serves web content to clients (browsers) over HTTP/HTTPS. When you type a URL in your browser, you're making a request to a web server.

How nginx Works as a Web Server

In this example, nginx serves the static HTML contact form directly to users:

server {
    listen 80;

    location / {
        root /usr/share/nginx/html;
        index index.html;
    }
}

Key Points:

• Nginx listens on port 80 for incoming HTTP requests

• Static files (HTML, CSS, JS) are served directly from /usr/share/nginx/html

• Clean, minimal configuration for serving static content

Advantages of Nginx as a Web Server:

• Speed: Handles static content incredibly fast

• Low Memory Usage: Uses an event-driven architecture

• Concurrent Connections: Can handle thousands of simultaneous connections

• Compression: Built-in gzip compression reduces bandwidth usage

Nginx as a Reverse Proxy

What is a Reverse Proxy?

A reverse proxy sits between clients and servers, forwarding client requests to appropriate backend servers and then returning the server's response back to the client. Unlike a forward proxy (which acts on behalf of clients), a reverse proxy acts on behalf of servers.

How reverse proxy use in this example

When submit the contact form, nginx forwards the request to our PHP back-ends:

# Proxy API requests to backend servers
location /api/ {
    # Remove /api prefix when forwarding to backend
    rewrite ^/api/(.*)$ /$1 break;

    # Proxy to upstream backend servers
    proxy_pass http://backend;

    # Headers for proper proxying
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

How it works:

1. Browser submits form to /api/contact.php

2. nginx receives the request

3. nginx removes /api prefix and forwards to backend

4. Backend processes the request

5. nginx returns the response to the browser

Benefits of Reverse Proxy:

• Security: Hides backend server details from clients

• SSL Termination: Handles encryption/decryption

• Caching: Can cache responses to improve performance

• Compression: Compresses responses before sending to clients

• Request Routing: Routes requests based on various criteria

Nginx as a Load Balancer

What is Load Balancing?

Load balancing distributes incoming network traffic across multiple servers to ensure no single server becomes overwhelmed. This improves application responsiveness and availability.

Load balancing in this example

Our Nginx configuration uses upstream servers for load balancing:

upstream backend_servers {
    server backend1:80;
    server backend2:80;
    # Explicitly use round-robin (default, but let's be clear)
}

Critical Configuration for Effective Load Balancing:

location /api/ {
    proxy_pass http://backend_servers/;
    # These settings ensure proper load balancing
    proxy_http_version 1.1;
    proxy_set_header Connection "";
}8

Why these settings matter:

• proxy_http_version 1.1: Forces HTTP/1.1 protocol

• proxy_set_header Connection "": Prevents connection reuse

• Result: Each request gets a fresh connection, ensuring visible round-robin distribution

Load Balancing Methods:

1. Round Robin (this example): Requests are distributed sequentially

2. Least Connections: Routes to server with fewest active connections

3. IP Hash: Routes based on client IP hash

4. Weighted: Assigns different weights to servers

More info: https://www.geeksforgeeks.org/system-design/load-balancing-algorithms/

Demonstrating Load Balancing

When you submit forms in our application, you'll notice:

• First submission → Backend 1 (shows container ID)

• Second submission → Backend 2 (shows different container ID)

• Third submission → Backend 1 (cycle repeats)

This demonstrates round-robin load balancing in action!

Apache vs nginx: Backend Processing

Apache Web Server

Apache uses a process-based or thread-based approach:

Apache Architecture:
┌─────────────┐
│   Request   │
└─────┬───────┘
      │
┌─────▼───────┐
│   Process/  │ ← Each request gets its own process/thread
│   Thread    │
└─────┬───────┘
      │
┌─────▼───────┐
│   PHP       │ ← PHP processes the request
│  Handler    │
└─────────────┘

Apache Characteristics:

• Process/Thread per connection: More memory usage

• Synchronous processing: Blocks while waiting for I/O

• Easier configuration: More straightforward for beginners

• Module system: Extensive module ecosystem

How Our PHP Back-ends Work with Apache

In the Docker setup, each backend container runs Apache with PHP:

# Our backends use php:8.1-apache image which includes:
# - Apache web server
# - PHP interpreter
# - mod_php module for Apache

The PHP code processes form data:

<?php
if ($_SERVER['REQUEST_METHOD'] === 'POST') {
    $name = htmlspecialchars($_POST['name'] ?? '');
    $email = htmlspecialchars($_POST['email'] ?? '');
    $server = gethostname(); // Shows which container processed it

    // Clean, inline HTML response with styling
    echo "<div style='font-family: Arial, sans-serif; max-width: 400px; margin: 50px auto; padding: 30px; background: #f8f9fa; border-radius: 10px; box-shadow: 0 2px 10px rgba(0,0,0,0.1);'>";
    echo "<h2 style='color: #28a745; text-align: center;'>✓ Message Sent Successfully!</h2>";
    echo "<p><strong>Name:</strong> $name</p>";
    echo "<p><strong>Email:</strong> $email</p>";
    echo "<p style='font-size: 12px; color: #666; text-align: center; margin-top: 20px;'>Processed by Backend Server: $server</p>";
    echo "</div>";
}
?>

Nginx vs Apache Performance

Running the Demo Application

Prerequisites

# Install Docker and Docker Compose
sudo apt update
sudo apt install docker.io docker-compose

# Add user to docker group
sudo usermod -aG docker $USER

Setup and Run

# Clone or create the project structure
mkdir simple-nginx-app && cd simple-nginx-app

# Create all files as shown in the first artifact
# Then run:
docker-compose up -d

# Access at: http://localhost:8080

Testing Load Balancing

1. Open http://localhost:8080

2. Fill out the contact form

3. Submit multiple times

4. Notice how container IDs alternate between submissions

5. This demonstrates round-robin load balancing!

Visual Demo: Round Robin Load Balancing in Action

Once the application is up and running, can observe the backend switching behavior using container IDs.

✅ Step 1: Check Running Containers

Run the following command to view active containers and their IDs:

docker ps

List of running containers with their names and container IDs

✅ Step 2: First Form Submission

Open the app in your browser at http://localhost:8080

1. Fill in the contact form with test values (e.g., Alice, alice@example.com)

2. Click Submit

Filled contact form before submission

Output showing “Message Sent Successfully!” along with a container ID (e.g., backend1)

✅ Step 3: Second Form Submission

Repeat the process with different values (e.g., Bob, bob@example.com):

Second form submission

Response showing success message with a different container ID (e.g., backend2)

✅ Step 4: Repeat to Observe Load Balancing

Each time you submit the form, you'll notice the container ID switches between backend1 and backend2. This confirms the round robin load balancing behavior configured in nginx.

Demonstration of backend switching between requests

Troubleshooting Common Issues

1. 502 Bad Gateway

• Backend servers are down

• Network connectivity issues

• Check docker-compose logs

2. Load Balancing Not Working

• Verify upstream configuration

• Check backend server status

• Review Nginx error logs

Conclusion

nginx's versatility as a web server, reverse proxy, and load balancer makes it an essential tool for modern web applications. This Docker example demonstrates these concepts in action:

• Web Server: Efficiently serves static HTML content

• Reverse Proxy: Routes API requests to backend services

• Load Balancer: Distributes traffic across multiple PHP back-ends using round-robin

The combination of Nginx with containerized back-ends provides a robust, scalable architecture that can handle high traffic while maintaining excellent performance. Whether you're building microservices, implementing high availability, or optimizing web application performance, understanding these concepts is crucial for modern web development.

Micro-services architecture is everywhere in modern software development, but getting started with Kubernetes can feel overwhelming. That's where K3s comes in - a lightweight Kubernetes distribution that's perfect for learning and development.

In this tutorial, I build and deploy a simple two-service application:

• User Service: Returns user information

• Greeting Service: Calls the user service and creates personalized greetings

Learning areas:

• Set up K3s on your server

• Containerize Node.js applications with Docker

• Deploy services to Kubernetes

• Automate deployments with GitLab CI/CD

GitLab Repository: https://gitlab.com/sachindu_personal/KubeGreet.git

✅ Step 1: Prerequisites and K3s Installation

Install K3s on Your Server

K3s is a lightweight Kubernetes distribution perfect for development and production environments.

# Install K3s
curl -sfL https://get.k3s.io | sh -

# Check node status (takes ~30 seconds)
sudo k3s kubectl get node

# Create kubectl symlink for easier access
sudo ln -s /usr/local/bin/k3s /usr/local/bin/kubectl

Expected output:

NAME          STATUS   ROLES                  AGE   VERSION
thinkcentre   Ready    control-plane,master   93s   v1.32.6+k3s1

Uninstall K3s (if needed)

/usr/local/bin/k3s-uninstall.sh

✅ Step 2: Building the Microservices Application

Project Structure

Create the following directory structure:

KubeGreet/
├── user-service/
│   ├── index.js
│   ├── package.json
|   ├── package-lock.json
│   ├── Dockerfile
│   └── k8s/
│       └── user-service-deployment.yaml
├── greeting-service/
│   ├── index.js
│   ├── package.json
|   ├── package-lock.json
│   ├── Dockerfile
│   └── k8s/
│       └── greeting-service-deployment.yaml
├── README.md
├──.gitignore
└──.gitlab-ci.yml

User Service Setup

1. Create user-service files (index.js and package.json)

2. Test locally:

    # Install all required packages listed in package.json
    npm install
   
    # Start the Node.js application
    npm start
   

3. Verify: Visit http://localhost:3000/user

Greeting Service Setup

1. Create greeting-service files (index.js and package.json)

2. Important: For local testing, modify the service URL:

    // Change from:const response = await axios.get('http://user-service:3000/user');// To:const response = await axios.get('http://localhost:3000/user');
   

3. Test locally:

    # Install all required packages listed in package.json
    npm install
   
    # Start the Node.js application
    npm start
   

4. Verify: Visit http://localhost:3001/greet

✅ Step 3: Containerizing with Docker

Install Docker (if not installed)

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# Verify installation
docker run hello-world

Build and Test Docker Images

User Service:

cd user-service
docker build -t username/user-service:latest .
docker run -d -p 3000:3000 --name testUserservice username/user-service:latest

Greeting Service:

cd greeting-service
docker build -t username/greeting-service:latest .
docker run -d -p 3001:3001 --name testGreetingservice username/greeting-service:latest

Push to Docker Hub

docker push username/user-service:latest
docker push username/greeting-service:latest

✅ Step 4: Setting up GitLab Repository

Initialize Git Repository

git init
git remote add origin https://gitlab.com/your-username/KubeGreet.git

# Create .gitignore
echo "node_modules/" >> .gitignore

git add .
git commit -m "initial commit"
git push --set-upstream origin main

GitLab Access Token

Create a personal access token for authentication:

• Path: Preferences > Access Token > Personal Access Token

✅ Step 5: Creating Kubernetes Manifests

📌 What is K3s and Why Use It?

K3s is a lightweight, easy-to-install Kubernetes distribution designed for development, edge computing, and IoT environments. It is simpler and consumes fewer resources compared to full Kubernetes, making it perfect for local testing and learning DevOps workflows.

📌 What is a Manifest File?

A Kubernetes manifest file is a YAML file that defines your Kubernetes resources, such as deployments, services, and config maps. It tells Kubernetes what to deploy and how to configure it.

For example, a deployment manifest specifies:

• Which Docker image to use

• Number of replicas

• Ports to expose

Manual Deployment to K3s on Localhost

Create deployment files:

• user-service/k8s/user-service-deployment.yaml

• greeting-service/k8s/greeting-service-deployment.yaml

Apply manifests:

kubectl apply -f user-service/k8s/user-service-deployment.yaml
kubectl apply -f greeting-service/k8s/greeting-service-deployment.yaml

Verify deployment:

kubectl get pods

Expected output:

NAME                                READY   STATUS    RESTARTS   AGE
greeting-service-576bbcfd76-zk2dg   1/1     Running   0          5m
user-service-6d5c9c4f79-r2fxl       1/1     Running   0          5m10s

Useful Commands

# Scale deployment
kubectl scale deployment user-service --replicas=3

# Restart service
kubectl rollout restart deployment user-service

# Delete pod
kubectl delete pod <pod-name>

✅ Step 6: Automated CI/CD Pipeline Setup

Step 1: Install GitLab Runner

# Install GitLab Runner
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
sudo apt-get update
sudo apt-get install gitlab-runner

# Verify installation
gitlab-runner --version

Step 2: Register GitLab Runner

Get Project Runner Token:

1. Go to your GitLab project

2. Navigate to Settings → CI/CD → Runners

3. Click "New project runner"

4. Configure runner settings:

   • Tags: k3s-deploy (or your preferred tag)

   • Description: K3s deployment runner

   • Maximum timeout: 3600 seconds (1 hour)

5. Click "Create runner"

6. Copy the registration token

Register the Runner:

sudo gitlab-runner register \
  --url https://gitlab.com \
  --token <project-runner-token> \
  --executor shell

During registration, you'll be prompted:

1. Description: Press Enter (uses default)

2. Tags: Press Enter (no tags needed)

3. Maintenance note: Press Enter (skip)

4. Executor: Type shell

Step 3: Start GitLab Runner Service

sudo gitlab-runner start
sudo systemctl enable gitlab-runner
sudo systemctl status gitlab-runner

Shared runner should stay on the running state, otherwise Docker image will not be created.

Step 4: Configure kubectl Access

# Set up kubectl for current user
mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config

# Test kubectl
kubectl get nodes

Step 5: Get Base64 Kubeconfig

cat ~/.kube/config | base64 -w 0

Copy the output for GitLab CI/CD variables.

Step 6: Configure GitLab CI/CD Variables

Go to Settings → CI/CD → Variables and add:

Step 7: Create Docker Registry Secret

kubectl create secret docker-registry gitlab-registry-secret \
  --docker-server=registry.gitlab.com \
  --docker-username=your-username \
  --docker-password=your-token \
  --docker-email=your-email@gmail.com

Step 8: Fix Runner Permissions

# Add gitlab-runner to sudo group
sudo usermod -a -G sudo gitlab-runner

# Fix K3s kubeconfig permissions
sudo chmod 644 /etc/rancher/k3s/k3s.yaml

# Alternative: Copy config for gitlab-runner
sudo mkdir -p /home/gitlab-runner/.kube
sudo cp /etc/rancher/k3s/k3s.yaml /home/gitlab-runner/.kube/config
sudo chown gitlab-runner:gitlab-runner /home/gitlab-runner/.kube/config
sudo chmod 600 /home/gitlab-runner/.kube/config

Step 9: Test Deployment

# Test manual deployment
kubectl apply -f user-service/k8s/user-service-deployment.yaml
kubectl get pods -l app=user-service

# Test the service
curl http://localhost:3000/greet

Step 10: Run CI/CD Pipeline

Now let's test the complete CI/CD pipeline by making a commit and verifying the automated deployment.

Trigger the Pipeline

Make a small change to test the automation:

# Commit and push
git add .
git commit -m "Test CI/CD pipeline"
git push origin main

Monitor Pipeline Execution

1. Go to GitLab: Navigate to your project → CI/CD → Pipelines

2. Check Pipeline Status: You should see a running pipeline with stages:

   • ✅ Build: Docker images are built and pushed

   • ✅ Deploy: Services are deployed to K3s cluster

Verify Successful Deployment

Check Pipeline Success:

# Check if pods are running
kubectl get pods

# Expected output:
NAME                                READY   STATUS    RESTARTS   AGE
greeting-service-576bbcfd76-zk2dg   1/1     Running   0          2m
user-service-6d5c9c4f79-r2fxl       1/1     Running   0          2m

Test the Application: Visit in your browser: http://<server-ip>:30081/greet

Expected Output:

⚠️ Common Errors and Troubleshooting

1. Pod not starting: Check logs with kubectl logs -l app=service-name

2. Permission denied: Ensure gitlab-runner has proper permissions

3. Registry pull errors: Verify registry secret is created correctly

Useful Commands

# Check pod status
kubectl describe pod -l app=user-service

# View logs
kubectl logs -l app=user-service

# Check services
kubectl get svc

💬 Tried this flow or need any step explained better? Drop a comment below – I welcome your feedback and let’s grow together as DevOps learners!

This project is a Python-based Email Subject Generator Application developed using the Flask framework. It follows a three-tier architecture, consisting of a front-end, back-end API, and database, each running as separate services to ensure modularity and scalability.

The tool enables users to input any scenario or context that requires an email. Based on the provided information, the application intelligently generates a suitable and impactful subject line for the email, enhancing communication clarity and professionalism.

To streamline development, testing, and deployment, we implemented separate CI and CD pipelines using Jenkins. The source code is hosted on Bit-bucket, and every push to the repository triggers an automated CI/CD workflow that:

• Runs code quality tests using SonarQube to ensure clean, maintainable, and secure code.

• Builds and tests the Flask application using Jenkins pipelines.

• Packages the app into a Docker image for consistent deployment across environments.

• Pushes the generated Docker image to Docker Hub.

• Optionally deploys the application to a target server or cloud platform via the CD pipeline.

Bit-bucket Repository Link: https://sachindumalshan@bitbucket.org/sachindu-work-space/email-subject-generator.git

✅ Step 1: Prerequisites and Environment Setup

Do the following on the server:

Install Docker

# Install docker original
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# Verify Docker installation:
docker run hello-world

Install Docker Compose

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

# Set the permission to execute Docker Compose:
sudo chmod +x /usr/local/bin/docker-compose

# Check Docker Compose version:
docker-compose --version

Native Jenkins Installation

# Update package lists and upgrade existing packages
sudo apt update && sudo apt upgrade -y

# Install OpenJDK 17 (required for Jenkins)
sudo apt install openjdk-17-jdk -y

# Verify Java installation
java -version

Download and add the Jenkins GPG key. This command adds Jenkins’ official GPG key to your system’s keyring so that APT trusts Jenkins packages.

# Download and add the Jenkins GPG key to the system keyrings
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee /usr/share/keyrings/jenkins-keyring.asc > /dev/null

Update package index and install Jenkins

# Update package index after adding Jenkins repository key
sudo apt update

# Install Jenkins
sudo apt install jenkins -y

# Check if port 8080 is already in use (Jenkins default port)
sudo netstat -tlnp | grep :8080

# Check current firewall status
sudo ufw status

# Allow incoming connections on Jenkins port (8080)
sudo ufw allow 8080

# Start Jenkins service
sudo systemctl start jenkins

# Enable Jenkins to start on boot
sudo systemctl enable jenkins

Access Jenkins from Another Computer

• Open the URL below in your browser. It will display the Jenkins startup interface and prompt you to enter the administrator password. Use the following command to retrieve the password, then enter it in the prompt to log into Jenkins.

# Use 'ifconfig' to get the server IP address
# Ex: http://192.168.8.129:8080/

http://<server-ip>:8080

💡 To get the Jenkins initial admin password, enter the following command on the Jenkins server:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

If you skip user configuration during setup:

• Username: admin

• Password: Use the output from above command (e.g. a5bba1b0c60d4edaadf420882fb12060)

Add Jenkins User to Docker Group and Verify Access

# Add the Jenkins user to the Docker group to allow it to run Docker commands
sudo usermod -aG docker jenkins

# Restart Jenkins to apply the new group permissions
sudo systemctl restart jenkins

# Verify that the Jenkins user can run Docker commands without permission errors
sudo -u jenkins docker ps

✅ Step 2: Prepare and Develop the Application

Project folder structure is here.

email-subject-generator/
├── docker-compose.yml
├── .env
├── .gitignore
├── README.md
├── sonar-project.properties
│
├── frontend/
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── app.py
│   ├── test_app.py
│   ├── templates/
│   │   └── index.html
│   └── static/
│       └── style.css
│
├── backend/
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── app.py
│   └── test_app.py
│
└── database/
    └── init.sql

📝 1. Create and Test the Frontend

Navigate to the frontend/ folder containing:

• Dockerfile

• requirements.txt

• app.py

• test_app.py

• templates/index.html

• static/style.css

All the coding files are available in the repository.

Install all dependencies:

pip install -r requirements.txt

To test locally, run:

python3 app.py

✔️ When running successfully, it will be available at:

# Paste the URL in the broswer
http://0.0.0.0:5000

📝 2. Create the .env File

email-subject-generator/
├── .env

The .env file is used to store environment-specific configuration details such as database credentials, API keys, ports, and other sensitive information required by the application.

• Why use a .env file?
  It helps keep configuration separate from code, allowing easy changes without modifying source files. This approach improves security and flexibility across different environments (development, testing, production).

• What is usually stored in a .env file?

  • Database host, user, password, and database name

  • Secret keys and tokens (e.g., API keys)

  • Application-specific settings like ports or debug flags

• Why is the .env file usually excluded from remote repositories?
  Because it contains sensitive data that should not be publicly exposed or shared, it is common practice to add .env to .gitignore. This prevents accidental leaks of credentials or secrets and keeps your application secure.

Note:
For learning purposes, a sample .env file has been included in the repository to demonstrate its structure and contents.
In real projects, avoid committing your actual .env files with sensitive data to public repositories.

📝 3. Create and Test the Backend and Database

Navigate to the backend/ and database/ folders:

• backend/

  • Dockerfile

  • requirements.txt

  • app.py

  • test_app.py

• database/

  • init.sql

All the coding files are available in the repository.

Install all dependencies:

pip install -r requirements.txt

To test locally, run:

python3 app.py

The backend will run on: http://0.0.0.0:5001

✅ Check if the server is listening

Run:

netstat -tuln | grep 5001

✔️ If you see output showing your Python/Flask process listening on port 5001, it confirms the server is running.

✅ Use curl to test endpoints

Run:

curl http://localhost:5001/api/health

✔️ You will see output similar to:

{
  "service": "email-subject-generator-backend",
  "status": "healthy",
  "timestamp": "2025-07-12T12:11:46.423913"
}

✔️ This confirms the API is working and returning the expected response.

✅ Step 3: Create Docker files and Docker Compose Configuration

💡 Note:

As per industry best practices, a Dockerfile is not needed for the database. Database containers can be configured directly using Docker Compose.

1. Dockerfile – Frontend

# Use the official Python 3.10.12 image with Alpine Linux (lightweight)
FROM python:3.10.12-alpine

# Set the working directory inside the container to /app
WORKDIR /app

# Copy only requirements.txt first to leverage Docker cache for faster builds when dependencies don't change
COPY requirements.txt .

# Install Python dependencies specified in requirements.txt
RUN pip install -r requirements.txt

# Copy the entire application code into the container's /app directory
COPY . .

# Expose port 5000 so it can be accessed externally if mapped
EXPOSE 5000

# Specify the default command to run the Flask app when the container starts
CMD ["python3", "app.py"]

Explanation: The EXPOSE command is used as good practice to document the port the container listens on.

If the build fails to pull the base image, login to Docker Hub:

docker login

✔️ Run the frontend container:

sudo docker run -d -p 5000:5000 --name email-gen-frontend emailgen-app:v1

2. Dockerfile – Backend

# Use the official Python 3.10.12 image with Alpine Linux (lightweight)
FROM python:3.10.12-alpine

# Set the working directory inside the container to /app
WORKDIR /app

# Copy only requirements.txt first to leverage Docker cache for faster builds when dependencies don't change
COPY requirements.txt .

# Install Python dependencies specified in requirements.txt
RUN pip install -r requirements.txt

# Copy the entire application code into the container's /app directory
COPY . .

# Expose port 5001 so it can be accessed externally if mapped
EXPOSE 5001

# Specify the default command to run the Flask app when the container starts
CMD ["python3", "app.py"]

✔️ Run the backend container:

sudo docker run -d -p 5001:5001 --name emailgen-be emailgen-backend:v1

Check running containers:

docker ps

# Or view all containers including stopped ones:
docker ps -a

# Example output:
CONTAINER ID   IMAGE                 COMMAND            CREATED          STATUS          PORTS                                       NAMES
3979a0bdfe82   emailgen-backend:v1   "python3 app.py"   6 seconds ago    Up 6 seconds    0.0.0.0:5001->5001/tcp, :::5001->5001/tcp   emailgen-be
a771f531d2f9   emailgen-app:v1       "python3 app.py"   14 minutes ago   Up 14 minutes   0.0.0.0:5000->5000/tcp, :::5000->5000/tcp   email-gen-frontend

3. Create and Run docker-compose.yml

version: '3.9'
services:
  frontend:
    # Use build for development, or set FRONTEND_IMAGE and IMAGE_TAG env vars for deployment
    build: ./frontend
    # image: ${FRONTEND_IMAGE:-emailgen-frontend}:${IMAGE_TAG:-latest}
    container_name: emailgen-frontend
    ports:
      - "5000:5000"
    depends_on:
      - backend
    environment:
      - BACKEND_URL=http://backend:5001
    networks:
      - emailgen-net
    restart: unless-stopped

  backend:
    # Use build for development, or set BACKEND_IMAGE and IMAGE_TAG env vars for deployment
    build: ./backend
    # image: ${BACKEND_IMAGE:-emailgen-backend}:${IMAGE_TAG:-latest}
    container_name: emailgen-backend
    ports:
      - "5001:5001"
    depends_on:
      - db
    environment:
      - MYSQL_HOST=db
      - MYSQL_PORT=3306
      - MYSQL_DATABASE=email_generator
      - MYSQL_USER=root
      - MYSQL_PASSWORD=rootpassword123
      - GROQ_API_KEY=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
    networks:
      - emailgen-net
    restart: unless-stopped

  db:
    image: mysql:8.0
    container_name: emailgen-db
    restart: unless-stopped
    environment:
      - MYSQL_ROOT_PASSWORD=rootpassword123
      - MYSQL_DATABASE=email_generator
    volumes:
      - ./database/init.sql:/docker-entrypoint-initdb.d/init.sql
      - mysql_data:/var/lib/mysql
    ports:
      - "3306:3306"
    networks:
      - emailgen-net
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-p=rootpassword123"]
      interval: 10s
      timeout: 5s
      retries: 5

volumes:
  mysql_data:

networks:
  emailgen-net:
    driver: bridge

After creating your docker-compose.yml file, you can run all services together with a single command.

Build and Run in detached mode:

To run in the background (detached mode), use:

docker-compose up -d --build

Check running containers

docker ps

You should see output similar to:

CONTAINER ID   IMAGE                 COMMAND            CREATED          STATUS          PORTS                                       NAMES
abcd1234efgh   emailgen-backend      "python3 app.py"   5 seconds ago    Up 5 seconds    0.0.0.0:5001->5001/tcp                     emailgen-backend
ijkl5678mnop   emailgen-frontend     "python3 app.py"   5 seconds ago    Up 5 seconds    0.0.0.0:5000->5000/tcp                     emailgen-frontend
qrst9012uvwx   mysql:8.0             "docker-entrypoi…" 5 seconds ago    Up 5 seconds    33060/tcp, 0.0.0.0:3306->3306/tcp          emailgen-db

Here you can see frontend, backend, and database containers running with their respective ports.

Test your application

• Open your browser and navigate to:

  • Frontend: http://localhost:5000

  • Backend API: http://localhost:5001

Confirm that your frontend connects with the backend and data saves/retrieves from the database successfully.

When done testing:

docker-compose down

This stops and removes containers, networks, and default volumes created by docker-compose up.

✅ Step 4: Push Code to Bit bucket Repository

# Initialize Git in your local project folder
git init

# Create .gitignore file to exclude node_modules and .env for security and clean repo
echo "node_modules/" >> .gitignore
echo ".env" >> .gitignore

# Track all changes and commit
git add .
git commit -m "Initial commit"

# Connect local repo to GitHub remote (initial setup)
git remote add origin https://sachindumalshan@bitbucket.org/sachindu-work-space/email-subject-generator.git

# Push code to main branch on GitHub
git branch -M main
git push -u origin main

❗ Bit bucket Push Error: Instead of Bit bucket password, use Access Token.

Create one from:
https://bitbucket.org/account/settings/app-passwords/ → Generate classic token with full access.

✅ Step 5: Set Up Jenkins

Go to: Manage Jenkins > Manage Plugins, Search and install the plugins as needed. Install essential plugins such as:

• Pipeline

• Git plugin

• Bitbucket Branch Source plugin

• SonarQube Scanner plugin

• Docker Pipeline plugin

• Parameterized Trigger plugin

• Pipeline Stage View Plugin

Ensure these plugins are installed and up-to-date before running the pipelines.

Configure Build Tools

1. JDK configure

# Name
JDK-17

# Set the JDK path as:
/usr/lib/jvm/java-17-openjdk-amd64

# check java jdk installation path
sudo update-alternatives --config java

If not found, Update and install OpenJDK 17:

sudo apt update
sudo apt install openjdk-17-jdk -y

2. Git configure

# Name
Default 

# Set the Git path as:
/usr/bin/git

# To check git path
which git

3. Docker configure

# Name
Docker

# Set the JDK path as:
/usr/bin/docker

Install and Configure SonarQube (Code Quality Analysis)

To run SonarQube analysis locally, you'll need to install both:

• SonarQube Server: The main server that processes and stores analysis results

• SonarQube Scanner: The client tool that analyzes your code and sends results to the server

Below is the installation guide for each.

Why use SonarQube?

• Ensures code quality, security, and maintainability by automatic static code analysis

• Detects bugs, vulnerabilities, code smells, and duplications early in the CI/CD pipeline

• Provides a dashboard for project code health with historical trends and actionable insights

1. Install SonarQube Server

# 1. Update system packages
sudo apt update

# 2. Install Java 17 (required for SonarQube)
sudo apt install -y openjdk-17-jdk

# 3. Verify Java installation
java -version

# 4. Create SonarQube user (recommended for security)
sudo useradd -r -s /bin/false sonarqube

# 5. Download SonarQube Community Edition
cd /opt
sudo wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-10.4.1.88267.zip

# 6. Extract SonarQube
sudo unzip sonarqube-10.4.1.88267.zip
sudo mv sonarqube-10.4.1.88267 sonarqube
sudo chown -R sonarqube:sonarqube /opt/sonarqube

# 7. Configure SonarQube (optional - edit if needed)
sudo nano /opt/sonarqube/conf/sonar.properties

# 8. Create systemd service file
sudo tee /etc/systemd/system/sonarqube.service > /dev/null <<EOF
[Unit]
Description=SonarQube service
After=syslog.target network.target

[Service]
Type=forking
ExecStart=/opt/sonarqube/bin/linux-x86-64/sonar.sh start
ExecStop=/opt/sonarqube/bin/linux-x86-64/sonar.sh stop
User=sonarqube
Group=sonarqube
Restart=always
LimitNOFILE=65536
LimitNPROC=4096

[Install]
WantedBy=multi-user.target
EOF

# 9. Reload systemd and enable SonarQube service
sudo systemctl daemon-reload
sudo systemctl enable sonarqube
sudo systemctl start sonarqube

# 10. Check SonarQube service status
sudo systemctl status sonarqube

# 11. Check if SonarQube is running (may take a few minutes to start)
echo "SonarQube is starting... Please wait 5-10 minutes."
echo "Access SonarQube at: http://localhost:9000"
echo "Default credentials: admin / admin"

2. Check SonarQube Status

# Check SonarQube service status
sudo systemctl status sonarqube

# Check running processes
ps -ef | grep sonarqube

# Check logs for startup or error details
sudo tail -f /opt/sonarqube/logs/sonar.log

3. Troubleshooting Common SonarQube Issues

If SonarQube is not working, run the script below to diagnose and resolve:

#!/bin/bash

# ===========================================
# Common SonarQube Issues and Fixes
# ===========================================

echo "=== Fix 1: If SonarQube is taking too long to start ==="
echo "Wait 5-10 minutes for first startup"
echo "Monitor logs: sudo tail -f /opt/sonarqube/logs/sonar.log"

echo "=== Fix 2: If there are memory issues ==="
free -h
sudo nano /opt/sonarqube/conf/sonar.properties
cat << 'EOF'
# Reduce memory usage
sonar.web.javaOpts=-Xms512m -Xmx1g
sonar.ce.javaOpts=-Xms512m -Xmx1g
sonar.search.javaOpts=-Xms512m -Xmx1g
EOF

echo "=== Fix 3: If Elasticsearch won't start ==="
sysctl vm.max_map_count
sudo sysctl -w vm.max_map_count=262144
echo 'vm.max_map_count=262144' | sudo tee -a /etc/sysctl.conf

echo "=== Fix 4: If there are permission issues ==="
sudo chown -R sonarqube:sonarqube /opt/sonarqube
id sonarqube

echo "=== Fix 5: If port 9000 is in use ==="
sudo lsof -i :9000
sudo nano /opt/sonarqube/conf/sonar.properties
# Change sonar.web.port=9001 if needed

echo "=== Fix 6: Restart SonarQube service ==="
sudo systemctl restart sonarqube
sudo systemctl status sonarqube

echo "=== Fix 7: Manual start if still not working ==="
sudo systemctl stop sonarqube
sudo -u sonarqube /opt/sonarqube/bin/linux-x86-64/sonar.sh start
sudo -u sonarqube /opt/sonarqube/bin/linux-x86-64/sonar.sh console

4. Install SonarQube Scanner

The scanner analyzes your code and sends results to the SonarQube server.

#!/bin/bash

# ===========================================
# Install SonarQube Scanner on Linux
# ===========================================

# 1. Download SonarQube Scanner
cd /opt
sudo wget https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-5.0.1.3006-linux.zip

# 2. Extract Scanner
sudo unzip sonar-scanner-cli-5.0.1.3006-linux.zip
sudo mv sonar-scanner-5.0.1.3006-linux sonar-scanner
sudo chown -R $(whoami):$(whoami) /opt/sonar-scanner

# 3. Add Scanner to PATH
echo 'export PATH=$PATH:/opt/sonar-scanner/bin' >> ~/.bashrc
source ~/.bashrc

# 4. Verify installation
sonar-scanner --version

# 5. Configure Scanner (optional)
sudo nano /opt/sonar-scanner/conf/sonar-scanner.properties

# Enter the URL in the browser and setup the SonarQube using dashboard
http://localhost:9000

# Username: admin
# Password: admin

🔐 Logging into SonarQube Web Dashboard

1. Open your browser and go to:
   http://localhost:9000

2. You will be prompted for login credentials.

   • Default username: admin

   • Default password: admin

3. Important: After first login, immediately change the password to something secure.

4. Next, you will be prompted to create a new project:

   • Enter a project name (e.g., Email Subject Generator)

   • Enter a project key (a unique identifier, e.g., email-subject-gen)

5. Choose whether the project will be connected as a local or remote project depending on your setup.

6. After configuration, you will be taken to the SonarQube project dashboard, where you can view metrics, code quality reports, and other insights.

Configure System

To integrate SonarQube scanning within your Jenkins pipeline, follow these steps to properly add SonarQube server details in Jenkins:

1. Open Jenkins dashboard and navigate to:
   Manage Jenkins → Configure System

2. Scroll down to the SonarQube servers section.

3. Click Add SonarQube.

4. Enter the following details:

   • Name: SonarQube (or any recognizable name)

   • Server URL: http://<your-sonarqube-server-ip>:9000 (e.g., http://localhost:9000)

   • Server authentication token:

     • Generate a token from SonarQube user profile → My Account → Security → Generate Tokens

     • Paste the generated token here to authenticate Jenkins with SonarQube.

5. Save the configuration.

6. Now, Jenkins can communicate with SonarQube to run code analysis as part of your CI pipeline.

Manage Credentials

Jenkins needs credentials to securely access services like Bitbucket, Docker Hub, and SonarQube. You add these credentials in Jenkins under Manage Jenkins > Manage Credentials.

• Bitbucket: Use your username and app password or token to allow Jenkins to clone and push code.

• Docker Hub: Provide your Docker username and access token so Jenkins can build and push images.

• SonarQube: Use a secret token from your SonarQube account for Jenkins to run code analysis.

Always store credentials securely in Jenkins and avoid hardcoding them in your code or pipeline scripts.

✅ Step 6: Setup CI Pipeline

Prerequisites Checklist

• Jenkins is installed and running

• Bitbucket repository is accessible and configured with Jenkins credentials

• Docker is installed and Jenkins user added to docker group

• SonarQube server is running and integrated with Jenkins (SonarQube plugin configured)

• Python and required build tools installed on Jenkins node (optional if you use Docker build)

Script for CI pipeline

pipeline {
    agent any

    // Add trigger configuration
    triggers {
        // Poll SCM every 2 minutes as fallback
        pollSCM('H/2 * * * *')
        // Bitbucket webhook trigger
        bitbucketPush()
    }

    environment {
        VENV = 'venv'
        DOCKER_HUB_USER = 'YOUR-DOCKER-HUB-USERNAME'
        IMAGE_TAG = "${BUILD_NUMBER}"
        FRONTEND_IMAGE = "${DOCKER_HUB_USER}/email-subject-generator-frontend"
        BACKEND_IMAGE = "${DOCKER_HUB_USER}/email-subject-generator-backend"
        DOCKER_HUB_CREDENTIALS = 'docker-hub-credentials'
        // SonarQube configuration
        SONAR_HOST_URL = 'http://localhost:9000'
        SONAR_PROJECT_KEY = 'email-subject-generator'
        SONAR_PROJECT_NAME = 'Email Subject Generator'
    }

    stages {
        stage('Clone Repository') {
            steps {
                script {
                    // Clone the repository
                    checkout([
                        $class: 'GitSCM',
                        branches: [[name: '*/main']],
                        userRemoteConfigs: [[
                            url: 'https://sachindumalshan@bitbucket.org/sachindu-work-space/email-subject-generator.git',
                            credentialsId: 'bitbucket-creds'
                        ]],
                        extensions: [
                            [$class: 'CleanBeforeCheckout'],
                            [$class: 'PruneStaleBranch']
                        ]
                    ])

                    // Set git commit hash after cloning
                    env.GIT_COMMIT_SHORT = sh(script: "git rev-parse --short HEAD", returnStdout: true).trim()
                    env.GIT_COMMIT_MESSAGE = sh(script: "git log -1 --pretty=%B", returnStdout: true).trim()
                    env.GIT_AUTHOR = sh(script: "git log -1 --pretty=%an", returnStdout: true).trim()

                    echo "Git commit: ${env.GIT_COMMIT_SHORT}"
                    echo "Commit message: ${env.GIT_COMMIT_MESSAGE}"
                    echo "Author: ${env.GIT_AUTHOR}"
                }
            }
        }

        stage('Set up Python Environment') {
            steps {
                sh '''
                    # Clean up any existing venv
                    rm -rf "$VENV"

                    # Install python3-venv if not available
                    if ! python3 -m venv --help > /dev/null 2>&1; then
                        echo "Installing python3-venv package..."
                        sudo apt update
                        sudo apt install -y python3-venv python3-pip
                    fi

                    # Create fresh virtual environment
                    python3 -m venv "$VENV"
                    . "$VENV/bin/activate"

                    # Upgrade pip
                    pip install --upgrade pip

                    # Install dependencies with error handling
                    if [ -f backend/requirements.txt ]; then
                        echo "Installing backend dependencies..."
                        pip install -r backend/requirements.txt
                    else
                        echo "backend/requirements.txt not found"
                        exit 1
                    fi

                    # Check if frontend has requirements.txt (optional)
                    if [ -f frontend/requirements.txt ]; then
                        echo "Installing frontend dependencies..."
                        pip install -r frontend/requirements.txt
                    else
                        echo "frontend/requirements.txt not found, skipping frontend Python dependencies"
                    fi

                    # Install coverage for Python code coverage
                    pip install coverage pytest pytest-cov

                    # Verify installation
                    pip list
                '''
            }
        }

        stage('Run Tests with Coverage') {
            steps {
                script {
                    try {
                        sh '''
                            . "$VENV/bin/activate"

                            # Set timeout for tests
                            timeout 300 bash -c '
                                # Create coverage reports directory
                                mkdir -p coverage-reports

                                # Run frontend tests with coverage
                                if [ -f frontend/app.py ]; then
                                    echo "Running frontend tests with coverage..."
                                    cd frontend
                                    coverage run --source=. -m pytest --junitxml=../coverage-reports/frontend-junit.xml . || python app.py --test
                                    coverage xml -o ../coverage-reports/frontend-coverage.xml
                                    coverage report
                                    cd ..
                                else
                                    echo "Frontend app.py not found, skipping frontend tests"
                                fi

                                # Run backend tests with coverage
                                if [ -f backend/app.py ]; then
                                    echo "Running backend tests with coverage..."
                                    cd backend
                                    coverage run --source=. -m pytest --junitxml=../coverage-reports/backend-junit.xml . || python app.py --test
                                    coverage xml -o ../coverage-reports/backend-coverage.xml
                                    coverage report
                                    cd ..
                                else
                                    echo "Backend app.py not found, skipping backend tests"
                                fi
                            '
                        '''
                    } catch (Exception e) {
                        echo "Tests failed: ${e.getMessage()}"
                        currentBuild.result = 'UNSTABLE'
                    }
                }
            }
        }

        stage('SonarQube Analysis') {
            steps {
                script {
                    try {
                        if (!fileExists('sonar-project.properties')) {
                        echo "sonar-project.properties file not found. Creating default configuration..."

                        // Create sonar-project.properties file
                        writeFile file: 'sonar-project.properties', text: """sonar.projectKey=${env.SONAR_PROJECT_KEY}
sonar.projectName=${env.SONAR_PROJECT_NAME}
sonar.projectVersion=${env.BUILD_NUMBER}
sonar.sources=frontend,backend
sonar.sourceEncoding=UTF-8
sonar.python.coverage.reportPaths=coverage-reports/frontend-coverage.xml,coverage-reports/backend-coverage.xml
sonar.python.xunit.reportPath=coverage-reports/frontend-junit.xml,coverage-reports/backend-junit.xml
sonar.exclusions=**/*.pyc,**/__pycache__/**,**/venv/**,**/node_modules/**,**/*.log,**/database/**,**/static/**,**/templates/**
sonar.tests=frontend,backend
sonar.test.inclusions=**/test_*.py,**/*test*.py
sonar.scm.provider=git
sonar.scm.revision=${env.GIT_COMMIT_SHORT}"""
                        }else{
                            echo "sonar-project.properties file found. Using existing configuration..."
                        }

                        // Run SonarQube analysis
                        withSonarQubeEnv('SonarQube') {
                            sh '''
                                echo "Running SonarQube analysis..."
                                echo "Project Key: $SONAR_PROJECT_KEY"
                                echo "SonarQube Host: $SONAR_HOST_URL"

                                # Use sonar-scanner with project file
                                sonar-scanner
                            '''
                        }

                    } catch (Exception e) {
                        echo "SonarQube analysis failed: ${e.getMessage()}"
                        currentBuild.result = 'UNSTABLE'
                    }
                }
            }
        }

        stage('Lightweight Code Quality') {
            steps {
                script {
                    try {
                        sh '''
                            . "$VENV/bin/activate"

                            echo "Running lightweight code quality checks..."

                            # Quick Python syntax check
                            echo "Checking Python syntax..."
                            python -m py_compile backend/app.py || echo "Backend syntax issues found"
                            python -m py_compile frontend/app.py || echo "Frontend syntax issues found"

                            # Quick linting with basic rules
                            echo "Running basic linting..."
                            pip install flake8 || true
                            flake8 --select=E9,F63,F7,F82 backend/ frontend/ || echo "Basic linting issues found"

                            # Check for common security issues
                            echo "Basic security check..."
                            grep -r "password.*=" backend/ frontend/ || echo "No hardcoded passwords found"

                            echo "Lightweight quality checks completed!"
                        '''
                    } catch (Exception e) {
                        echo "Lightweight quality checks failed: ${e.getMessage()}"
                        echo "Continuing anyway..."
                    }

                    // Always continue - never fail the pipeline
                    echo "SonarQube analysis running in background at: ${env.SONAR_HOST_URL}/dashboard?id=${env.SONAR_PROJECT_KEY}"
                }
            }
        }

        stage('Build Docker Images') {
            steps {
                script {
                    // Build frontend image
                    if (fileExists('frontend/Dockerfile')) {
                        sh '''
                            echo "Building frontend Docker image..."
                            echo "Dockerfile content:"
                            cat frontend/Dockerfile
                            echo "---"

                            # Force rebuild without cache to ensure fresh build
                            docker build --no-cache -t "$FRONTEND_IMAGE:$IMAGE_TAG" -t "$FRONTEND_IMAGE:latest" ./frontend
                        '''
                    } else {
                        echo "Frontend Dockerfile not found, skipping frontend build"
                    }

                    // Build backend image
                    if (fileExists('backend/Dockerfile')) {
                        sh '''
                            echo "Building backend Docker image..."
                            echo "Dockerfile content:"
                            cat backend/Dockerfile
                            echo "---"

                            # Force rebuild without cache to ensure fresh build
                            docker build --no-cache -t "$BACKEND_IMAGE:$IMAGE_TAG" -t "$BACKEND_IMAGE:latest" ./backend
                        '''
                    } else {
                        echo "Backend Dockerfile not found, skipping backend build"
                    }
                }
            }
        }

        stage('Test Docker Images') {
            steps {
                script {
                    try {
                        // Test frontend image
                        if (sh(script: "docker images -q \"\$FRONTEND_IMAGE:\$IMAGE_TAG\"", returnStdout: true).trim()) {
                            sh '''
                                echo "Testing frontend Docker image..."

                                # Clean up any existing test containers
                                docker stop frontend-test 2>/dev/null || true
                                docker rm frontend-test 2>/dev/null || true

                                # Find an available port for frontend
                                PORT1=5000
                                while netstat -tulpn | grep -q ":$PORT1 "; do
                                    PORT1=$((PORT1 + 2))
                                done
                                echo "Using port $PORT1 for frontend testing..."

                                # Run frontend container in background
                                docker run -d --name frontend-test -p $PORT1:5000 "$FRONTEND_IMAGE:$IMAGE_TAG"

                                # Wait for container to start
                                echo "Waiting for frontend container to start..."
                                sleep 15

                                # Check if container is running
                                if docker ps | grep -q frontend-test; then
                                    echo "Frontend container is running successfully on port $PORT1"

                                    # Test if the application is responding
                                    echo "Testing frontend health..."
                                    timeout 30 bash -c "until curl -f http://localhost:$PORT1 >/dev/null 2>&1; do sleep 2; done" || echo "Frontend health check timeout - this might be normal if no health endpoint exists"

                                    # Check container logs for any obvious errors
                                    echo "Frontend container logs:"
                                    docker logs frontend-test --tail 20
                                else
                                    echo "Frontend container failed to start"
                                    docker logs frontend-test
                                    exit 1
                                fi

                                # Clean up
                                docker stop frontend-test
                                docker rm frontend-test
                            '''
                        } else {
                            echo "Frontend image not found, skipping frontend test"
                        }

                        // Test backend image
                        if (sh(script: "docker images -q \"\$BACKEND_IMAGE:\$IMAGE_TAG\"", returnStdout: true).trim()) {
                            sh '''
                                echo "Testing backend Docker image..."

                                # Clean up any existing test containers
                                docker stop backend-test 2>/dev/null || true
                                docker rm backend-test 2>/dev/null || true

                                # Find an available port for backend
                                PORT2=5001
                                while netstat -tulpn | grep -q ":$PORT2 "; do
                                    PORT2=$((PORT2 + 2))
                                done
                                echo "Using port $PORT2 for backend testing..."

                                # Run backend container in background
                                docker run -d --name backend-test -p $PORT2:5001 "$BACKEND_IMAGE:$IMAGE_TAG"

                                # Wait for container to start
                                echo "Waiting for backend container to start..."
                                sleep 15

                                # Check if container is running
                                if docker ps | grep -q backend-test; then
                                    echo "Backend container is running successfully on port $PORT2"

                                    # Test if the application is responding
                                    echo "Testing backend health..."
                                    timeout 30 bash -c "until curl -f http://localhost:$PORT2 >/dev/null 2>&1; do sleep 2; done" || echo "Backend health check timeout - this might be normal if no health endpoint exists"

                                    # Check container logs for any obvious errors
                                    echo "Backend container logs:"
                                    docker logs backend-test --tail 20
                                else
                                    echo "Backend container failed to start"
                                    docker logs backend-test
                                    exit 1
                                fi

                                # Clean up
                                docker stop backend-test
                                docker rm backend-test
                            '''
                        } else {
                            echo "Backend image not found, skipping backend test"
                        }

                    } catch (Exception e) {
                        echo "Image tests failed: ${e.getMessage()}"
                        currentBuild.result = 'UNSTABLE'

                        // Clean up on failure
                        sh '''
                            docker stop frontend-test backend-test 2>/dev/null || true
                            docker rm frontend-test backend-test 2>/dev/null || true
                        '''
                    }
                }
    }
}

        stage('Push Docker Images') {
            when {
                // Only push if quality gate passed and build is successful
                anyOf {
                    expression { currentBuild.result == null }
                    expression { currentBuild.result == 'SUCCESS' }
                }
            }
            steps {
                script {
                    try {
                        echo "Pushing Docker images to Docker Hub..."
                        withDockerRegistry([credentialsId: 'docker-hub-credentials', url: 'https://index.docker.io/v1/']) {
                            sh '''
                                # Push frontend images
                                echo "Pushing frontend images..."
                                docker push "$FRONTEND_IMAGE:$IMAGE_TAG"
                                docker push "$FRONTEND_IMAGE:latest"

                                # Push backend images
                                echo "Pushing backend images..."
                                docker push "$BACKEND_IMAGE:$IMAGE_TAG"
                                docker push "$BACKEND_IMAGE:latest"
                            '''
                        }
                        echo "Successfully pushed all Docker images"
                    } catch (Exception e) {
                        echo "Failed to push Docker images: ${e.getMessage()}"
                        throw e
                    }
                }
            }
        }

        stage('Trigger CD Pipeline') {
            when {
                anyOf {
                        expression { currentBuild.result == null }
                        expression { currentBuild.result == 'SUCCESS' }
                    }
            }
            steps {
                script {
                    try {
                        echo "Triggering CD pipeline for automatic deployment..."

                        // Get the current build number to use as image tag
                        def imageTag = env.BUILD_NUMBER

                        // Trigger CD pipeline for dev environment automatically
                        build job: 'email-gen-app-cd', 
                              parameters: [
                                  string(name: 'IMAGE_TAG', value: imageTag),
                                  booleanParam(name: 'SKIP_TESTS', value: false)
                              ],
                              wait: false  // Don't wait for CD to complete

                        echo "CD pipeline triggered successfully"
                        echo "Image tag: ${imageTag}"
                        echo "Frontend image: ${env.FRONTEND_IMAGE}:${imageTag}"
                        echo "Backend image: ${env.BACKEND_IMAGE}:${imageTag}"
                        echo "CD pipeline will deploy to ports - Frontend: 5000, Backend: 5001, DB: 3306"

                    } catch (Exception e) {
                        echo "Failed to trigger CD pipeline: ${e.getMessage()}"
                        echo "You can manually trigger the CD pipeline with the following parameters:"
                        echo "- IMAGE_TAG: ${env.BUILD_NUMBER}"
                        echo "- SKIP_TESTS: false"
                        echo "Images available for deployment:"
                        echo "- Frontend: ${env.FRONTEND_IMAGE}:${env.BUILD_NUMBER}"
                        echo "- Backend: ${env.BACKEND_IMAGE}:${env.BUILD_NUMBER}"
                        // Don't fail the CI pipeline if CD trigger fails
                    }
                }
            }
        }
    }

    post {
        always {
            script {
                // Ensure we're in a node context for cleanup
                try {
                    // Clean up virtual environment
                    sh '''
                        if [ -d "$VENV" ]; then
                            rm -rf "$VENV"
                            echo "Cleaned up virtual environment"
                        fi
                    '''

                    // Clean up SonarQube working directory
                    sh '''
                        if [ -d ".sonar" ]; then
                            rm -rf .sonar
                            echo "Cleaned up SonarQube working directory"
                        fi
                    '''

                    // Clean up coverage reports
                    sh '''
                        if [ -d "coverage-reports" ]; then
                            rm -rf coverage-reports
                            echo "Cleaned up coverage reports"
                        fi
                    '''

                    // Clean up Docker images to save space
                    sh '''
                        echo "Cleaning up Docker images..."
                        docker image prune -f

                        # Clean up any remaining test containers
                        docker stop frontend-test backend-test 2>/dev/null || true
                        docker rm frontend-test backend-test 2>/dev/null || true
                    '''
                } catch (Exception e) {
                    echo "Cleanup failed: ${e.getMessage()}"
                }
            }
        }
        success {
            echo 'Pipeline completed successfully!'
            echo "Frontend image: ${env.FRONTEND_IMAGE}:${env.IMAGE_TAG}"
            echo "Backend image: ${env.BACKEND_IMAGE}:${env.IMAGE_TAG}"
            echo "SonarQube analysis completed. Check ${env.SONAR_HOST_URL}/dashboard?id=${env.SONAR_PROJECT_KEY}"
            echo "Triggered by commit: ${env.GIT_COMMIT_SHORT} by ${env.GIT_AUTHOR}"
            echo "Commit message: ${env.GIT_COMMIT_MESSAGE}"
        }
        failure {
            echo 'Pipeline failed!'
            echo "Failed commit: ${env.GIT_COMMIT_SHORT} by ${env.GIT_AUTHOR}"
            echo "Commit message: ${env.GIT_COMMIT_MESSAGE}"
        }
        unstable {
            echo 'Pipeline completed but tests or quality gate failed!'
            echo "Unstable commit: ${env.GIT_COMMIT_SHORT} by ${env.GIT_AUTHOR}"
        }
    }
}

Instructions to Create and Test the Pipeline

1. Login to Jenkins dashboard.

2. Click “New Item”.

   • Enter name: email-gen-ci

   • Select Pipeline, click OK.

3. Scroll to Pipeline section at bottom.

4. Paste the above Groovy script.

5. Configure the following credentials and environment values in Jenkins:

   • bitbucket-credentials-id – your Bitbucket username/password or token ID

   • dockerhub-credentials-id – your Docker Hub username/password credentials ID

   • Replace your-dockerhub-username with your actual Docker Hub username

   • Replace Bitbucket repo URL with your repository URL

   • SonarQubeServer – configured name under Manage Jenkins > Configure System > SonarQube servers

6. Click “Save”.

7. Click “Build Now” to test the pipeline.

Verify the CI Pipeline

• Build starts and shows each stage in Blue Ocean or classic view

• Checkout pulls your Bitbucket code

• Code Quality Analysis stage executes SonarQube scan and updates Sonar dashboard

• Build stage builds Docker image

• Test stage runs container and executes pytest tests inside

• Push Docker Image stage pushes image to Docker Hub

✅ Step 7: Setup CD Pipeline

pipeline {
    agent any

    parameters {
        string(
            name: 'IMAGE_TAG',
            defaultValue: 'latest',
            description: 'Docker image tag to deploy (e.g., latest, 123, v1.0.0)'
        )
        booleanParam(
            name: 'SKIP_TESTS',
            defaultValue: false,
            description: 'Skip deployment tests'
        )
    }

    environment {
        DOCKER_HUB_USER = 'YOUR-DOCKER-HUB-USERNAME'
        FRONTEND_IMAGE = "${DOCKER_HUB_USER}/email-subject-generator-frontend"
        BACKEND_IMAGE = "${DOCKER_HUB_USER}/email-subject-generator-backend"
        DOCKER_HUB_CREDENTIALS = 'docker-hub-credentials'

        // Fixed ports for single deployment
        FRONTEND_PORT = '5000'
        BACKEND_PORT = '5001'
        DB_PORT = '3306'
    }

    stages {
        stage('Validate Parameters') {
            steps {
                script {
                    echo "=== Deployment Configuration ==="
                    echo "Image Tag: ${params.IMAGE_TAG}"
                    echo "Skip Tests: ${params.SKIP_TESTS}"
                    echo "Frontend Image: ${env.FRONTEND_IMAGE}:${params.IMAGE_TAG}"
                    echo "Backend Image: ${env.BACKEND_IMAGE}:${params.IMAGE_TAG}"
                    echo "Ports - Frontend: ${env.FRONTEND_PORT}, Backend: ${env.BACKEND_PORT}, DB: ${env.DB_PORT}"
                }
            }
        }

        stage('Clone Repository') {
            steps {
                script {
                    checkout([
                        $class: 'GitSCM',
                        branches: [[name: '*/main']],
                        userRemoteConfigs: [[
                            url: 'https://sachindumalshan@bitbucket.org/sachindu-work-space/email-subject-generator.git',
                            credentialsId: 'bitbucket-creds'
                        ]],
                        extensions: [
                            [$class: 'CleanBeforeCheckout']
                        ]
                    ])

                    echo "Repository cloned successfully"
                }
            }
        }

        stage('Prepare Environment') {
            steps {
                script {
                    // Check if docker-compose.yml file exists
                    if (!fileExists('docker-compose.yml')) {
                        echo "docker-compose.yml file not found. Creating default configuration..."

                        // Create deployment-ready docker-compose file
                        def composeContent = """version: '3.9'

        services:
          frontend:
            image: ${env.FRONTEND_IMAGE}:${params.IMAGE_TAG}
            container_name: emailgen-frontend
            ports:
              - "${env.FRONTEND_PORT}:5000"
            depends_on:
              - backend
            environment:
              - BACKEND_URL=http://backend:5001
            networks:
              - emailgen-net
            restart: unless-stopped

          backend:
            image: ${env.BACKEND_IMAGE}:${params.IMAGE_TAG}
            container_name: emailgen-backend
            ports:
              - "${env.BACKEND_PORT}:5001"
            depends_on:
              - db
            environment:
              - MYSQL_HOST=db
              - MYSQL_PORT=3306
              - MYSQL_DATABASE=email_generator
              - MYSQL_USER=root
              - MYSQL_PASSWORD=rootpassword123
              - GROQ_API_KEY=YOUR-GROQ-API-KEY
            networks:
              - emailgen-net
            restart: unless-stopped

          db:
            image: mysql:8.0
            container_name: emailgen-db
            restart: unless-stopped
            environment:
              - MYSQL_ROOT_PASSWORD=rootpassword123
              - MYSQL_DATABASE=email_generator
            volumes:
              - ./database/init.sql:/docker-entrypoint-initdb.d/init.sql
              - mysql_data:/var/lib/mysql
            ports:
              - "${env.DB_PORT}:3306"
            networks:
              - emailgen-net
            healthcheck:
              test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-p=rootpassword123"]
              interval: 10s
              timeout: 5s
              retries: 5

        volumes:
          mysql_data:

        networks:
          emailgen-net:
            driver: bridge
        """

                        // Write the compose file
                        writeFile file: "docker-compose.yml", text: composeContent
                        echo "Docker-compose file created with deployment configuration"

                    } else {
                        echo "docker-compose.yml file found. Using existing configuration..."

                        // Check if existing compose file uses build or image
                        def composeContent = readFile('docker-compose.yml')

                        if (composeContent.contains('build:') && !composeContent.contains('image:')) {
                            echo "Existing compose file uses 'build' configuration."
                            echo "Setting environment variables for potential image override..."

                            // Set environment variables that can be used if needed
                            env.FRONTEND_IMAGE = env.FRONTEND_IMAGE ?: 'emailgen-frontend'
                            env.BACKEND_IMAGE = env.BACKEND_IMAGE ?: 'emailgen-backend'
                            env.IMAGE_TAG = params.IMAGE_TAG ?: 'latest'

                            echo "Environment variables set:"
                            echo "FRONTEND_IMAGE: ${env.FRONTEND_IMAGE}"
                            echo "BACKEND_IMAGE: ${env.BACKEND_IMAGE}"
                            echo "IMAGE_TAG: ${env.IMAGE_TAG}"
                        }

                        // Display existing compose file content for verification
                        echo "Current docker-compose.yml content:"
                        sh 'cat docker-compose.yml'
                    }
                }
            }
        }

        stage('Pull Docker Images') {
            steps {
                script {
                    try {
                        echo "Pulling Docker images..."
                        withDockerRegistry([credentialsId: 'docker-hub-credentials', url: 'https://index.docker.io/v1/']) {
                            sh """
                                echo "Pulling frontend image..."
                                docker pull ${env.FRONTEND_IMAGE}:${params.IMAGE_TAG}

                                echo "Pulling backend image..."
                                docker pull ${env.BACKEND_IMAGE}:${params.IMAGE_TAG}

                                echo "Pulling MySQL image..."
                                docker pull mysql:8.0
                            """
                        }
                        echo "All images pulled successfully"
                    } catch (Exception e) {
                        echo "Failed to pull images: ${e.getMessage()}"
                        throw e
                    }
                }
            }
        }

        stage('Stop Previous Deployment') {
            steps {
                script {
                    try {
                        echo "Stopping previous deployment..."
                        sh """
                            # Stop and remove existing containers
                            docker-compose down --remove-orphans || true

                            # Clean up any dangling containers
                            docker stop emailgen-frontend emailgen-backend emailgen-db 2>/dev/null || true
                            docker rm emailgen-frontend emailgen-backend emailgen-db 2>/dev/null || true

                            echo "Previous deployment stopped successfully"
                        """
                    } catch (Exception e) {
                        echo "Warning: Failed to stop previous deployment: ${e.getMessage()}"
                        echo "This might be normal if it's the first deployment"
                    }
                }
            }
        }

        stage('Deploy Application') {
            steps {
                script {
                    try {
                        echo "Deploying application..."
                        sh """
                            # Deploy using docker-compose
                            docker-compose up -d

                            echo "Deployment initiated successfully"

                            # Wait for services to start
                            echo "Waiting for services to start..."
                            sleep 30

                            # Check if containers are running
                            echo "Checking container status..."
                            docker-compose ps
                        """
                    } catch (Exception e) {
                        echo "Deployment failed: ${e.getMessage()}"
                        throw e
                    }
                }
            }
        }

        stage('Health Check') {
            when {
                expression { !params.SKIP_TESTS }
            }
            steps {
                script {
                    try {
                        echo "Performing health checks..."
                        sh """
                            # Wait for applications to be ready
                            echo "Waiting for applications to be ready..."
                            sleep 45

                            # Check if containers are running
                            echo "=== Container Status ==="
                            docker ps | grep emailgen || echo "No containers found"

                            # Check container logs for errors
                            echo "=== Backend Logs ==="
                            docker logs emailgen-backend --tail 20 || echo "Cannot get backend logs"

                            echo "=== Frontend Logs ==="
                            docker logs emailgen-frontend --tail 20 || echo "Cannot get frontend logs"

                            echo "=== Database Logs ==="
                            docker logs emailgen-db --tail 20 || echo "Cannot get database logs"

                            # Debug database connection
                            echo "=== Database Connection Debug ==="
                            docker exec emailgen-backend printenv | grep MYSQL || echo "No MYSQL env vars found"

                            echo "=== Testing Database ==="
                            docker exec emailgen-db mysql -u root -prootpassword123 -e "SHOW DATABASES;" || echo "Database connection failed"

                            # Test application endpoints
                            echo "=== Health Check Tests ==="

                            # Test backend health endpoint
                            echo "Testing backend health at http://localhost:${env.BACKEND_PORT}/api/health"
                            timeout 60 bash -c 'until curl -f http://localhost:${env.BACKEND_PORT}/api/health >/dev/null 2>&1; do sleep 5; done' || echo "Backend health check failed"

                            # Test frontend
                            echo "Testing frontend at http://localhost:${env.FRONTEND_PORT}"
                            timeout 60 bash -c 'until curl -f http://localhost:${env.FRONTEND_PORT} >/dev/null 2>&1; do sleep 5; done' || echo "Frontend health check failed"

                            # Test database connection
                            echo "Testing database connection..."
                            timeout 30 bash -c 'until docker exec emailgen-db mysqladmin ping -h localhost --silent; do sleep 2; done' || echo "Database health check failed"

                            echo "Health checks completed"
                        """
                    } catch (Exception e) {
                        echo "Health check failed: ${e.getMessage()}"
                        currentBuild.result = 'UNSTABLE'
                    }
                }
            }
        }

        stage('Deployment Verification') {
            steps {
                script {
                    try {
                        echo "Verifying deployment..."
                        sh """
                            # Final verification
                            echo "=== Final Deployment Status ==="
                            docker-compose ps

                            # Check if all services are up
                            RUNNING_SERVICES=\$(docker-compose ps --services --filter "status=running" | wc -l)
                            TOTAL_SERVICES=\$(docker-compose ps --services | wc -l)

                            echo "Running services: \$RUNNING_SERVICES/\$TOTAL_SERVICES"

                            if [ "\$RUNNING_SERVICES" -eq "\$TOTAL_SERVICES" ]; then
                                echo "✅ All services are running successfully"
                            else
                                echo "❌ Some services are not running"
                                exit 1
                            fi
                        """
                    } catch (Exception e) {
                        echo "Deployment verification failed: ${e.getMessage()}"
                        throw e
                    }
                }
            }
        }
    }

    post {
        success {
            echo "=== Deployment Successful ==="
            echo "Image Tag: ${params.IMAGE_TAG}"
            echo "Frontend URL: http://localhost:${env.FRONTEND_PORT}"
            echo "Backend URL: http://localhost:${env.BACKEND_PORT}"
            echo "Database Port: ${env.DB_PORT}"
            echo "=== Services ==="
            script {
                try {
                    sh "docker-compose ps"
                } catch (Exception e) {
                    echo "Could not display final service status"
                }
            }
        }
        always {
            script {
                try {
                    // Clean up unused images
                    sh """
                        docker image prune -f
                        echo "Cleaned up unused images"
                    """
                } catch (Exception e) {
                    echo "Cleanup failed: ${e.getMessage()}"
                }
            }
        }
        failure {
            echo "=== Deployment Failed ==="
            echo "Image Tag: ${params.IMAGE_TAG}"
            echo "Check the logs above for details"

            script {
                try {
                    // Show container logs on failure
                    sh """
                        echo "=== Container Logs on Failure ==="
                        docker logs emailgen-backend --tail 30 2>/dev/null || echo "No backend logs"
                        docker logs emailgen-frontend --tail 30 2>/dev/null || echo "No frontend logs"
                        docker logs emailgen-db --tail 30 2>/dev/null || echo "No database logs"
                    """
                } catch (Exception e) {
                    echo "Could not retrieve container logs"
                }
            }
        }
        unstable {
            echo "=== Deployment Completed with Issues ==="
            echo "The application was deployed but some health checks failed"
            echo "Please check the application manually"
        }
    }
}

Instructions to Create CD Pipeline

1. Login to Jenkins dashboard

2. Click “New Item”

3. Enter name: email-gen-cd

4. Select Pipeline, click OK

5. Scroll down to Pipeline section

6. Paste your provided pipeline Groovy script (above)

7. Click “Save”

Final Verification

1. Access application:

   • Webapp URL: http://<server-ip>:5000

   • Backend URL: http://<server-ip>:5001

2. Check running containers:

    docker ps
   

✅ Step 8: Configure Bit bucket Web-hook with Jenkins

Setting up a webhook enables automatic triggering of pipelines when changes are pushed to the repository. In this project, pushing or updating the main branch:

• Runs the CI pipeline automatically

• After CI completion, triggers the CD pipeline

• Deploys the updated application seamlessly

Instructions

1. Go to your Bitbucket repository.

2. Click on Repository settings in the sidebar.

3. Under Workflow, click Webhooks.

4. Click “Add Webhook”.

5. Configure the webhook with the following:

• Title: Jenkins CI/CD Trigger

• URL: http://:8080/bitbucket-hook/

• Triggers: Repository push

6. Save the webhook configuration.

Verify Webhook Functionality

If the webhook is configured correctly:

• A successful POST request status (200) will display in Bitbucket webhook page after pushing code.

• Jenkins will show a new CI pipeline build triggered automatically.

• Upon CI pipeline success, the CD pipeline will start automatically, deploying the application.

✅ Step 9: Update Repository with New Application – Email Subject Generator (Gen AI App)

After verifying that the CI/CD pipeline works correctly with your basic multi-tier setup, we now update the repository with the actual production application code – the Email Subject Generator (Gen AI App).

🚀 What’s Being Deployed Now?

This new version of the app uses Generative AI techniques to analyze the scenario input and generate a suitable subject line for emails.

Application Features

• Generate Email Subject

• View History

❗Common Errors & Fixes

Below is a list of the key issues encountered during the CI/CD setup and deployment of the Email Subject Generator (Gen AI App), along with the solutions applied.

❌ Error 1: (1045) Access denied for user 'sachindu'@'localhost'

🔑 Cause: User does not exist or lacks privileges.

✅ Solution A – Create New User

sudo mysql -u root -p

CREATE USER 'sachindu'@'localhost' IDENTIFIED BY 'your_password';
GRANT ALL PRIVILEGES ON email_generator.* TO 'sachindu'@'localhost';
FLUSH PRIVILEGES;

✅ Update .env

MYSQL_USER=sachindu
MYSQL_PASSWORD=your_password

✅ Update app.py database configuration accordingly.

✅ Solution B – Use Root User

sudo mysql -u root -p

GRANT ALL PRIVILEGES ON email_generator.* TO 'root'@'localhost';
FLUSH PRIVILEGES;

✅ Update .env

MYSQL_USER=root
MYSQL_PASSWORD=rootpassword123

❌ Error 2: MySQL Authentication Issues (caching_sha2_password)

🔑 Cause: MySQL 8.0 uses incompatible default auth plugin.

✅ Solution: Ensure .env and docker-compose use correct root credentials only. Use mysql_native_password if needed.

❌ Error 3: CI/CD – Container Running Error (Backend can't connect to DB)

🔑 Cause: Database not correctly connected with backend container.

✅ Solution:

• Verify MYSQL_HOST, USER, PASSWORD, DATABASE in .env.

• Ensure database container is healthy and accessible before backend starts.

• Confirm docker-compose depends_on is configured for backend -> db.

❌ Error 4: SonarQube Installation Issues

🔑 Cause: Online installer fails or times out in some environments.

✅ Solution: Use offline local testing method, such as:

• Download and extract SonarQube manually.

• Start locally using:

./bin/linux-x86-64/sonar.sh start

• Access via localhost:9000 for analysis testing.

💬 Feedback & Suggestions

I always welcome your feedback and suggestions to improve these projects and pipelines further.
✨ Love to hear your thoughts!

Are you looking to integrate powerful AI capabilities into your development and testing projects without breaking the bank? Groq offers an excellent solution for developers who want to leverage fast language models in their applications at zero cost. In this comprehensive guide, we'll walk you through everything you need to know about using Groq in your development workflow.

What is Groq?

Groq is a cutting-edge AI inference platform that provides lightning-fast language model processing. It's designed to deliver exceptional performance for AI applications, making it an ideal choice for developers working on chatbots, content generation, code analysis, and testing automation.

Getting Started with Groq

Step 1: Creating Your Groq Account

First, you'll need to access the Groq platform:

1. Visit the Official Website: Navigate to groq.com

2. Account Creation: You'll be prompted to create an account or log in using your existing Google or GitHub credentials

3. Choose Your Preferred Method: Select either Google or GitHub for quick authentication

Step 2: Exploring the Groq Dashboard

Once logged in, you'll see the Groq dashboard with several key features:

1. User Message Testing: Test the AI capabilities by entering any prompt message to see how the model responds

2. Template Script Access: Use the dropdown menu to access pre-built template scripts

3. Model Selection: Choose from various available models, keeping in mind that free access may have different limitations depending on the model selected

Step 3: Creating Your Groq API Key

To use Groq in your development projects, you'll need to generate an API key:

1. Navigate to API Keys: From the dashboard, locate and click on the "API Keys" section

2. Create New Key: Click the "Create API Key" button

3. Name Your Key: Give your API key a preferred name (e.g., "Development Project" or "Testing Environment")

4. Generate Key: Click "Submit" to create your API key

5. Save Your Secret Key: You'll receive a secret key - copy and save it immediately as you won't be able to view it again

⚠️ Important Security Note: Store your API key securely and never share it publicly or commit it to version control systems.

Setting Up Groq in Your Development Project

Prerequisites

The following demonstration focuses on creating a Python project, but Groq also supports JavaScript, CURL, and JSON implementations. Before integrating Groq into your project, ensure you have:

Note: Check the official Groq documentation for language-specific examples.

Here the below demonstration given by assuming creating a python project. It can use as JavaScript, curl, JSON Before integrating Groq into your project, ensure you have:

• Python installed on your system

• A valid Groq API key

• Basic understanding of Python programming

Step 1: Installation

Install the Groq library using pip:

pip install groq

Step 2: Basic Implementation

Here's the basic template provided by Groq for Python projects:

import os
from groq import Groq

client = Groq(
    api_key=os.environ.get("GROQ_API_KEY"),
)

chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "Explain the importance of fast language models",
        }
    ],
    model="llama-3.3-70b-versatile",
)

print(chat_completion.choices[0].message.content)

Step 3: Configuration

Important Configuration Steps:

1. Set Your API Key: Ensure your GROQ_API_KEY environment variable is properly configured

2. Customize Content: Replace the content field with your specific prompt message

3. Choose Your Model: Select the appropriate model based on your project requirements

Practical Implementation Examples

Example 1: Content Generation for Testing

import os
from groq import Groq

client = Groq(api_key=os.environ.get("GROQ_API_KEY"))

def generate_test_data():
    chat_completion = client.chat.completions.create(
        messages=[
            {
                "role": "user",
                "content": "Generate 5 sample user profiles for testing an e-commerce application",
            }
        ],
        model="llama-3.3-70b-versatile",
    )
    return chat_completion.choices[0].message.content

# Use in your testing workflow
test_data = generate_test_data()
print(test_data)

Example 2: Code Review Assistant

def code_review_assistant(code_snippet):
    chat_completion = client.chat.completions.create(
        messages=[
            {
                "role": "user",
                "content": f"Review this code for potential improvements and bugs: {code_snippet}",
            }
        ],
        model="llama-3.3-70b-versatile",
    )
    return chat_completion.choices[0].message.content

Advantages of Using Groq

1. Zero Cost: Perfect for development and testing phases

2. Fast Processing: Lightning-fast inference speeds

3. Multiple Models: Access to various language models

4. Easy Integration: Simple API integration

5. Comprehensive Documentation: Well-documented APIs and examples

Conclusion

Groq provides an excellent opportunity for developers to integrate powerful AI capabilities into their projects without any upfront costs. By following this guide, you can quickly get started with Groq and begin leveraging its capabilities in your development and testing workflows.

Whether you're building automated testing tools, generating content, or creating AI-powered applications, Groq offers the speed and reliability you need to succeed.

What's Next?

Start by creating your Groq account and experimenting with the basic examples provided in this guide. As you become more comfortable with the platform, explore advanced features and integration possibilities.

Have you used Groq in your development projects? Share your experiences and suggestions in the comments below! Your feedback helps the community learn and grow together.

Why Build a Home-lab?

• Learn Linux system administration and networking

• Gain practical DevOps skills (beyond theory!)

• Host and experiment with personal projects

• Control your environment, privately and securely

What I Used

Step-by-Step Setup

1. Clean the Old Laptop

• Removed unnecessary parts like the broken display

• Replaced thermal paste and ensured good airflow

• Built a custom enclosure for better cooling (Here I used old plastic box)

2. Install Ubuntu Server

• Download Ubuntu Server 24.04.2 : https://ubuntu.com/download/server

  

• Create a bootable USB with Balena Etcher or Rufus

  • Rufus download Link: https://rufus.ie/en/

  • Balena Etcher download Link: https://etcher.balena.io/

• Install using minimal setup (no GUI) to save resources

  

https://www.youtube.com/watch?v=K2m52F0S2w8

• Connect the laptop to your router via Ethernet for a stable connection.

• Install basic networking tools:

  sudo apt install net-tools
  

  Use ifconfig to check your local IP address:

  ifconfig
  

✅ Verify SSH Access

Try accessing your server from another device:

ssh username@your-local-ip

• Replace username and your-local-ip with your actual credentials (you can find the IP from ifconfig under inet).

• If it is successful, show like below.

  

❗ Troubleshooting SSH

If SSH is not working, follow these steps:

1. Check if SSH is installed:

   dpkg -l | grep openssh-server
   

2. If not installed, install it:

   sudo apt install openssh-server
   

3. Start and enable the SSH service:

   sudo systemctl start ssh
   sudo systemctl enable ssh
   

4. Check SSH status:

   sudo systemctl status ssh
   

🔒 Tip: Allow SSH in your firewall (if UFW is enabled):

sudo ufw allow ssh

Absolutely! Here's the updated Point 4 and 5 in markdown format, with the additional notes on remote access, account authentication, and accessing from other devices:

4. Set Up Remote Access with Tailscale

• Install Tailscale on the server machine (a simple, secure VPN for remote access):

  curl -fsSL https://tailscale.com/install.sh | sh
  sudo tailscale up
  

• Sign in using your Tailscale account to register the server : https://tailscale.com/

  

• Important:
  When setting up, use the same Tailscale account across:

  • Your server (homelab laptop)

  • Any other device (e.g., personal laptop, phone) you use to access it

✅ Tailscale Tip:
If the setup is correct, you will see a “Connected” status with a green dot next to your device in the Tailscale dashboard when it’s online.

• After setup, access your homelab from anywhere:

  ssh username@your-tailscale-ip
  

  Find the Tailscale IP by running:

  tailscale ip -4
  

5. Basic Server Configuration & Security

• Update the system:

  sudo apt update && sudo apt upgrade
  

• Install essential tools:

  sudo apt install htop git ufw fail2ban
  

• Set up firewall with UFW:

  sudo ufw allow OpenSSH
  sudo ufw enable
  

• Optional but recommended: Enable fail2ban to protect against brute-force attacks:

  sudo systemctl enable fail2ban
  sudo systemctl start fail2ban
  

💬 Got a Better Way?

If you’ve set up your own home-lab differently or have suggestions to improve this setup, I’d love to hear from you!

💡 Share your thoughts:

• Did you use another remote access method like OpenVPN, or WireGuard?

• Have tips to optimize performance on low-spec hardware?

• Got questions or need help with a similar setup?

👇 Drop a comment below - let’s learn from each other!

This project is a Node JS based SEO Content Optimizer Tool designed to analyze and enhance web content for better visibility on search engines. The tool provides keyword analysis, readability checks, and metadata suggestions to help developers and marketers create high-ranking content.

To streamline development and deployment, we implemented a fully automated CI/CD pipeline using Jenkins. The source code is hosted on GitHub, and every push to the repository triggers an automated workflow that:

1. Builds and tests the Node.js application using Jenkins.

2. Packages the app into a Docker image.

3. Pushes the image to Docker Hub.

4. Optionally deploys the image to a target server or cloud platform.

GitHub Repository Link: https://github.com/sachindumalshan/node-appp.git

Step 1: Prerequisites

Install Docker (Original)

# Install docker original
sudo apt install docker.io -y

# Verify by running hello-world container
sudo docker run hello-world

Native Jenkins Installation

Instead of Docker-based Jenkins, install Jenkins natively:

sudo apt update && sudo apt upgrade -y
sudo apt install openjdk-17-jdk -y

curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee /usr/share/keyrings/jenkins-keyring.asc > /dev/null

sudo apt update
sudo apt install jenkins -y

sudo systemctl start jenkins
sudo systemctl enable jenkins
sudo ufw allow 8080

⚠️ Note: Installing Java JDK 17 separately is not sufficient for the latest Jenkins version. It recommends using Java JDK 21.

Install Git

sudo apt install git

Step 2: Setting Up Jenkins

Access Jenkins from Another Computer

• Open the URL below in your browser. It will display the Jenkins startup interface and prompt you to enter the administrator password. Use the following command to retrieve the password, then enter it in the prompt to log into Jenkins.

# Use 'ifconfig' to get the server IP address
# Ex: http://192.168.8.129:8080/

http://<server-ip>:8080

Get the Password:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

# If the above command doesn't work, try:
docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword

If you skip adding user details during setup, you can log in using the default admin account.

• Username: admin

• Password: Use the password generated using the command shown above.

Then, install the necessary libraries and complete the configurations. You will end up on the Jenkins dashboard.

Install Necessary Plugins

Go to: Manage Jenkins > Manage Plugins

Search and install the plugins as needed. Install essential plugins such as:

• Git

• Pipeline

• NodeJS

• Other relevant tools

Configure Build Tools

1. JDK configure

# Name
JDK-17

# Set the JDK path as:
/usr/lib/jvm/java-17-openjdk-amd64

# check java jdk installation path
sudo update-alternatives --config java

If not found, Update and install OpenJDK 17:

sudo apt update
sudo apt install openjdk-17-jdk -y

2. Git configure

# Name
Default 

# Set the Git path as:
/usr/bin/git

# To check git path
which git

3. Node JS configure

• Name - Node JS-18

• ✅ check the box Install automatically

• Select the Node JS version

4. Docker configure

# Name
Docker

# Set the JDK path as:
/usr/bin/docker

Step 3: Prepare the Application

Create Sample Node.js Application

# Create a new directory for your Node.js application
mkdir my-node-app

# Move into the newly created directory
cd my-node-app

# Initialize a new Node.js project with default settings
# This creates a package.json file with default values
npm init -y

Create index.js:

import http from "http";
import fs from "fs";
import path from "path";
import { fileURLToPath } from "url";

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

const server = http.createServer((req, res) => {
  const filePath = path.join(__dirname, "message.txt");

  fs.readFile(filePath, "utf8", (err, data) => {
    if (err) {
      res.writeHead(500, { "Content-Type": "text/plain" });
      res.end("Error reading file");
    } else {
      res.writeHead(200, { "Content-Type": "text/plain" });
      res.end(data);
    }
  });
});

server.listen(3000, () => {
  console.log("Server is running on port 3000");
});

Create message.txt:

Hello, this is a Node.js application without Express!

Run:

node index.js

❗ ERROR: To load an ES module, set "type": "module" in package.json.

Fix

Add to package.json:

"type": "module",

Step 4: Push to GitHub

# Initialize Git in the Project Folder
git init

# Create a `.gitignore` File
echo "node_modules/" >> .gitignore
echo ".env" >> .gitignore

# Track Changes and Commit
git add .
git commit -m "Initial commit"

# Connect Local Repository to Remote
git remote add origin https://github.com/sachindumalshan/node-appp.git

# Push Code to the Main Branch
git branch -M main
git push -u origin main

❗ GitHub Push Error: Instead of GitHub password, use Access Token.

Create one from:
https://github.com/settings/tokens → Generate classic token with full access.

Step 5: Jenkins Pipeline Configuration

Create new pipeline in Jenkins:

pipeline {
    agent any
    
    // GitHub webhook trigger
    triggers {
        githubPush()
    }
    
    environment {
        DOCKER_REGISTRY = 'docker.io'
        DOCKER_IMAGE = 'example/node-app'
        DOCKER_TAG = "${BUILD_NUMBER}"
        DOCKER_CREDENTIALS_ID = 'docker-hub-credentials'
        APP_PORT = '5000'
        CONTAINER_NAME = 'node-app-container'
    }
    
    stages {
        stage('Clone Repository') {
            steps {
                // Use the GitHub token for authentication
                checkout scmGit(
                    branches: [[name: '*/main']], 
                    extensions: [], 
                    userRemoteConfigs: [[
                        credentialsId: 'githubtoken', 
                        url: 'https://github.com/sachindumalshan/node-appp.git'
                    ]]
                )
            }
        }
        
        stage('Build') {
            steps {
                sh 'echo "Building the application..."'
                sh 'ls -la'  // Show what files we have
            }
        }
        
        stage('Test') {
            steps {
                sh 'echo "Running tests..."'
            }
        }
        
        stage('Code Quality') {
            steps {
                sh 'echo "Running code quality checks..."'
                // Add linting, security scanning, etc.
                // sh 'npm run lint || echo "No lint script found, skipping..."'
            }
        }
        
        stage('Docker Build') {
            steps {
                script {
                    // Build with both latest and build number tags
                    echo "Building Docker image: \({DOCKER_IMAGE}:\){DOCKER_TAG}"
                    sh "docker build -t \({DOCKER_IMAGE}:\){DOCKER_TAG} -t ${DOCKER_IMAGE}:latest ."
                }
            }
        }
        
        stage('Docker Push') {
            steps {
                script {
                        sh "docker login -u DockerHub_username -p DockerHub_Password"
                        sh "docker push \({DOCKER_IMAGE}:\){DOCKER_TAG}"
                        sh "docker push ${DOCKER_IMAGE}:latest"
                        sh "docker logout"
                }
            }
        }
        
        stage('Pre-Deploy Cleanup') {
            steps {
                script {
                    echo "🧹 Starting cleanup process..."
                    
                    sh '''
                        set +e  # Don't exit on errors - we want to continue cleanup
                        
                        echo "=== Cleanup Process Started ==="
                        
                        # Method 1: Direct container removal (most reliable)
                        echo "1. Removing container: ${CONTAINER_NAME}"
                        
                        # Check if container exists (running or stopped)
                        if docker ps -a --format '{{.Names}}' | grep -q "^\({CONTAINER_NAME}\)"; then
                            echo "   📦 Container ${CONTAINER_NAME} found, removing..."
                            
                            # Stop the container first (ignore errors if already stopped)
                            echo "   🛑 Stopping container..."
                            sudo docker stop ${CONTAINER_NAME} 2>/dev/null || echo "   ℹ️ Container might already be stopped"
                            
                            # Wait a moment
                            sleep 2
                            
                            # Force remove the container
                            echo "   🗑️ Removing container..."
                            sudo docker rm -f ${CONTAINER_NAME} 2>/dev/null || echo "   ⚠️ Failed to remove container"
                            
                            # Wait a moment for cleanup
                            sleep 2
                            
                        else
                            echo "   ℹ️ No container named ${CONTAINER_NAME} found"
                        fi
                        
                        # Method 2: Kill any process using the port
                        echo "2. Checking port ${APP_PORT} usage..."
                        
                        # Find process using the port
                        PORT_PID=\((sudo netstat -tlnp 2>/dev/null | grep ":\){APP_PORT} " | awk '{print $7}' | cut -d'/' -f1 | head -1)
                        
                        if [ -n "\(PORT_PID" ] && [ "\)PORT_PID" != "-" ] && [ "$PORT_PID" != "" ]; then
                            echo "   ⚠️ Found process \(PORT_PID using port \){APP_PORT}"
                            echo "   💀 Killing process..."
                            sudo kill -9 "$PORT_PID" 2>/dev/null || echo "   ℹ️ Process might already be dead"
                            sleep 2
                        else
                            echo "   ✅ Port ${APP_PORT} appears to be free"
                        fi
                        
                        # Method 3: Final verification and cleanup
                        echo "3. Final verification..."
                        
                        # Double-check container is gone
                        if docker ps -a --format '{{.Names}}' | grep -q "^\({CONTAINER_NAME}\)"; then
                            echo "   ⚠️ Container still exists, attempting nuclear removal..."
                            
                            # Get container ID
                            CONTAINER_ID=\((docker ps -a --format '{{.ID}}' --filter name=\){CONTAINER_NAME})
                            if [ -n "$CONTAINER_ID" ]; then
                                echo "   🎯 Container ID: $CONTAINER_ID"
                                
                                # Try to get and kill the main process
                                MAIN_PID=\((sudo docker inspect "\)CONTAINER_ID" 2>/dev/null | grep '"Pid"' | head -1 | awk -F': ' '{print $2}' | tr -d ',' | tr -d ' ' 2>/dev/null || echo "")
                                
                                if [ -n "\(MAIN_PID" ] && [ "\)MAIN_PID" != "0" ] && [ "$MAIN_PID" != "null" ]; then
                                    echo "   💀 Killing main process PID: $MAIN_PID"
                                    sudo kill -9 "$MAIN_PID" 2>/dev/null || echo "   ℹ️ PID might already be dead"
                                    sleep 2
                                fi
                                
                                # Force remove with container ID
                                echo "   🗑️ Force removing by ID: $CONTAINER_ID"
                                sudo docker rm -f "$CONTAINER_ID" 2>/dev/null || echo "   ⚠️ Failed to remove by ID"
                                sleep 2
                            fi
                        fi
                        
                        # Method 4: Clean up any orphaned containers
                        echo "4. Cleaning up orphaned containers..."
                        sudo docker container prune -f 2>/dev/null || echo "   ℹ️ Container prune completed"
                        
                        # Final wait
                        sleep 3
                        
                        # Verification
                        echo "\\n=== Cleanup Verification ==="
                        if docker ps -a --format '{{.Names}}' | grep -q "^\({CONTAINER_NAME}\)"; then
                            echo "   ❌ ERROR: Container ${CONTAINER_NAME} still exists!"
                            echo "   📋 Existing containers:"
                            docker ps -a --format "table {{.Names}}\\t{{.Image}}\\t{{.Status}}" | grep ${CONTAINER_NAME} || true
                            echo "   🚨 This will cause deployment to fail!"
                        else
                            echo "   ✅ Container ${CONTAINER_NAME} successfully removed"
                        fi
                        
                        # Check port
                        if sudo netstat -tlnp 2>/dev/null | grep -q ":${APP_PORT} "; then
                            echo "   ⚠️ WARNING: Port ${APP_PORT} still in use"
                            sudo netstat -tlnp 2>/dev/null | grep ":${APP_PORT} " || true
                        else
                            echo "   ✅ Port ${APP_PORT} is available"
                        fi
                        
                        echo "\\n🎉 Cleanup process completed!"
                        
                        # Always exit with success to continue pipeline
                        exit 0
                    '''
                }
            }
        }
        
        stage('Deploy'){
            steps {
                script {
                    echo "🚀 Deploying application..."
                    
                    // Additional safety check before deployment
                    sh '''
                        echo "=== Pre-deployment Safety Check ==="
                        
                        # Final check if container still exists
                        if docker ps -a --format '{{.Names}}' | grep -q "^\({CONTAINER_NAME}\)"; then
                            echo "❌ FATAL: Container ${CONTAINER_NAME} still exists!"
                            echo "Attempting emergency removal..."
                            
                            # Emergency removal
                            docker stop ${CONTAINER_NAME} 2>/dev/null || true
                            sleep 2
                            docker rm -f ${CONTAINER_NAME} 2>/dev/null || true
                            sleep 2
                            
                            # Check again
                            if docker ps -a --format '{{.Names}}' | grep -q "^\({CONTAINER_NAME}\)"; then
                                echo "❌ CRITICAL: Unable to remove existing container!"
                                echo "Manual intervention required."
                                exit 1
                            fi
                        fi
                        
                        echo "✅ Safety check passed - ready for deployment"
                    '''
                    
                    // Wait a moment after cleanup
                    sleep 2
                    
                    // Deploy with correct port mapping
                    sh """
                        echo "Starting new container: ${CONTAINER_NAME}"
                        docker run -d \\
                            --name ${CONTAINER_NAME} \\
                            --restart unless-stopped \\
                            -p \({APP_PORT}:\){APP_PORT} \\
                            \({DOCKER_IMAGE}:\){DOCKER_TAG}
                        
                        echo "✅ Container started successfully"
                    """
                    
                    // Verify container is running
                    sh """
                        echo "=== Deployment Verification ==="
                        docker ps --format "table {{.Names}}\\t{{.Image}}\\t{{.Status}}\\t{{.Ports}}" | grep ${CONTAINER_NAME} || {
                            echo "❌ Container not found in running state!"
                            docker ps -a | grep ${CONTAINER_NAME} || echo "Container not found at all!"
                            exit 1
                        }
                        
                        echo "Container logs (first 20 lines):"
                        docker logs --tail 20 ${CONTAINER_NAME}
                    """
                }
            }
        }
        
        stage('Health Check') {
            steps {
                script {
                    echo "🏥 Performing health check..."
                    
                    sh """
                        echo "Waiting for application to start..."
                        sleep 10
                        
                        echo "Health check attempts:"
                        SUCCESS=false
                        
                        for i in 1 2 3 4 5; do
                            echo "Attempt \$i/5..."
                            
                            # Check if container is still running
                            if ! docker ps | grep -q ${CONTAINER_NAME}; then
                                echo "❌ Container ${CONTAINER_NAME} is not running!"
                                docker logs ${CONTAINER_NAME}
                                exit 1
                            fi
                            
                            # Check HTTP endpoint
                            if curl -f -s --max-time 10 http://localhost:${APP_PORT} > /dev/null; then
                                echo "✅ Health check passed on attempt \$i!"
                                SUCCESS=true
                                break
                            else
                                echo "❌ Health check failed on attempt \$i"
                                if [ \$i -lt 5 ]; then
                                    echo "Waiting 10 seconds before retry..."
                                    sleep 10
                                fi
                            fi
                        done
                        
                        if [ "\$SUCCESS" = "false" ]; then
                            echo "❌ All health checks failed!"
                            echo "=== Debug Information ==="
                            docker ps | grep ${CONTAINER_NAME} || echo "Container not running"
                            docker logs --tail 50 ${CONTAINER_NAME}
                            exit 1
                        fi
                        
                        echo "🎉 Application is healthy and running!"
                    """
                }
            }
        }
    }
    
    post {
        always {
            // Clean up Docker images to save space (but keep recent ones)
            sh 'docker image prune -f --filter "until=24h"'
        }
        success {
            echo '✅ Pipeline completed successfully!'
            echo "🚀 Application is accessible at: http://100.82.148.95:${APP_PORT}"
            
            // Show deployment summary
            sh """
                echo "=== Deployment Summary ==="
                echo "Image: \({DOCKER_IMAGE}:\){DOCKER_TAG}"
                echo "Container: ${CONTAINER_NAME}"
                echo "Port: ${APP_PORT}"
                echo "Build: ${BUILD_NUMBER}"
                echo "Time: \$(date)"
                echo "Status: \\((docker inspect --format='{{.State.Status}}' \){CONTAINER_NAME})"
                echo "=========================="
            """
        }
        failure {
            echo '❌ Pipeline failed!'
            // Enhanced debugging information
            sh '''
                echo "=== Failure Debug Information ==="
                echo "Docker containers (all):"
                docker ps -a --format "table {{.Names}}\\t{{.Image}}\\t{{.Status}}\\t{{.Ports}}" || true
                
                echo "\\nPort usage:"
                netstat -tlnp 2>/dev/null | grep ":5000 " || echo "No process using port 5000"
                
                echo "\\nContainer logs (if exists):"
                docker logs --tail 50 ${CONTAINER_NAME} 2>/dev/null || echo "No container logs available"
                
                echo "\\nDocker system info:"
                docker system df || true
                echo "========================"
            '''
        }
        cleanup {
            // Clean workspace
            cleanWs()
        }
    }
}

Then, click the "Build Now" button. This will start the pipeline build process and display the build status.

Step 6: Setting Up GitHub Web hook with Jenkins

Prerequisites

• Jenkins must be publicly accessible (not behind a private IP or Tailscale-only address).

• Jenkins pipeline is already configured with your GitHub repo.

• Webhook endpoint URL (replace with your actual IP or domain):

http://<server-ip>:8080/github-webhook/

💡 If you're using Tailscale, expose Jenkins using Tailscale Funnel:

sudo tailscale funnel 8080

This will give a public URL like:

https://yourname.ts.net/github-webhook/

🪝 Steps to Add GitHub Web hook

1. Go to your GitHub repository.

2. Navigate to: Settings → Web-hooks → Add web-hook

3. In the Payload URL, enter:

   http://<your-server-ip>:8080/github-webhook/
   

   (Use your Tail-scale Funnel URL if applicable)

4. Set Content type to:

   application/json
   

5. Choose:

   • Just the push event

6. Click Add web hook

Step 7: Update Repository with New Application – SEO Content Optimizer Tool (Gen AI App)

After verifying the Jenkins pipeline works with your basic Node.js app, we now replace it with the actual Gen AI application – a SEO Content Optimizer Tool.

🚀 What's Being Deployed?

This new version of the app uses Generative AI techniques to analyze and optimize content for better SEO performance. It provides:

• Keyword extraction

• Meta tag suggestions

• Readability analysis

• Title and description optimization

• AI-based content suggestions

📦 Steps to Update the Application Code

1. Replace Code Locally
   Update your project folder (my-node-app/) with the files for the SEO Optimizer Tool.

   Make sure it includes your Dockerfile, updated index.js, and any other required modules like openai, axios, etc.

2. Install New Dependencies (if applicable)

   npm install

❗Common Errors & Fixes

Below is a list of the key issues encountered during the CI/CD setup and deployment of the SEO Content Optimizer tool, along with the solutions applied.

🔐 1. Jenkins Initial Admin Password Not Found

Problem:
Jenkins prompts for an initial admin password but the default command didn’t work.

Default command:

sudo cat /var/jenkins_home/secrets/initialAdminPassword

✅ Fix: If Jenkins runs as a Docker container, use:

docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword

☕ 2. JDK Path Not Found for Jenkins Build Tools

Problem:
Unable to configure JDK path under Jenkins build tool settings.

✅ Fix:

sudo apt update
sudo apt install openjdk-17-jdk -y

Set the path in Jenkins as:

/usr/lib/jvm/java-17-openjdk-amd64

📦 3. Node.js App Not Starting

Problem:
Server failed to start with this error:

Error [ERR_REQUIRE_ESM]: Must use import to load ES Module

✅ Fix:
Add the following to your package.json:

"type": "module",

Then re-run:

node index.js

🔐 4. GitHub Push Authentication Failed

Problem:
While pushing code to GitHub, password authentication failed.

✅ Fix:
Use a Personal Access Token (PAT) instead of your GitHub account password.

Steps to create a token:

• Go to GitHub Token Settings

• Click Generate new token (classic)

• Select required scopes (repo access)

• Copy the token and use it as password in the terminal

🐳 5. Docker Commands Failed in Jenkins Pipeline

Problem:
Jenkins pipeline fails during Docker stages with messages like:

docker: command not found
or
permission denied

Root Cause:
Jenkins was using the Snap Docker version while the system used APT Docker, causing conflicts. Jenkins had no permission to access the host Docker daemon.

✅ Fix:

1. Uninstall both versions:

   sudo snap remove docker
   sudo apt remove docker docker.io
   

2. Install Docker cleanly via APT:

   sudo apt install docker.io -y
   

3. Add Jenkins to Docker group:

   sudo usermod -aG docker jenkins
   sudo systemctl restart jenkins
   

4. Verify access:

   sudo -u jenkins docker ps
   

🔄 6. Jenkins Container Doesn’t Persist After Reboot

Problem:
After rebooting the server, the app is no longer accessible.
No container is running.

✅ Fix Options:

• Use the --restart unless-stopped flag when starting containers in Jenkins:

docker run -d --name node-app-container --restart unless-stopped -p 5000:5000 <image-name>

• Ensure Docker starts on boot:

sudo systemctl enable docker

• Optionally add a health check stage to re-deploy if the container isn't running.