Building a Production-Ready Cloud-Native Microservice with Complete CI/CD Pipeline on AWS EKS

📋 Project Overview

This comprehensive guide walks you through building a production-grade cloud-native microservice using FastAPI, Docker, Kubernetes (AWS EKS), and GitHub Actions CI/CD. You'll learn how to implement enterprise-level DevOps practices including automated testing, container orchestration, auto-scaling, health monitoring, and deployment notifications.

What You'll Build:

• Automated CI/CD pipeline using GitHub Actions

• Kubernetes deployment on AWS EKS with auto-scaling

• Multi-environment setup (dev, staging, production)

• Production observability and notifications

Tech Stack: Python, FastAPI, Docker, Kubernetes, AWS EKS, ECR, GitHub Actions, Prometheus, n8n

GitHub Repository: https://github.com/sachindumalshan/cloud-native-microservice-pipeline-monitor

---

📑 Table of Contents

1. Prerequisites

2. Phase 1: Local Development Setup

3. Phase 2: Containerization with Docker

4. Phase 3: Version Control & CI Setup

5. Phase 4: AWS Infrastructure Setup

6. Phase 5: Kubernetes Deployment

7. Phase 6: Production Reliability Features

8. Phase 7: Observability & Monitoring

9. Phase 8: Multi-Environment Deployment

10. Phase 9: Deployment Notifications

11. Common Errors & Solutions

12. Conclusion & Key Takeaways

---

Prerequisites

Before starting, ensure you have:

• Python 3.12+ installed

• Docker Desktop running

• AWS Account with appropriate permissions

• GitHub account

• Basic knowledge of Python, REST APIs, and command line

• kubectl and AWS CLI installed

---

Phase 1: Local Development Setup

Step 1.1: Create Project Structure

mkdir health_metrics_service
cd health_metrics_service
python3 -m venv venv
source venv/bin/activate

Step 1.2: Install Dependencies

pip install fastapi uvicorn pytest

Step 1.3: Create FastAPI Application

Create app.py:

from fastapi import FastAPI
import random, time

app = FastAPI()

@app.get("/health")
def health_check():
    return {"status": "healthy"}

@app.get("/metrics")
def metrics():
    cpu_load = random.uniform(0, 100)
    memory_usage = random.uniform(0, 100)
    return {"cpu": cpu_load, "memory": memory_usage}

@app.post("/simulate_load")
def simulate_load(duration: int = 5):
    start = time.time()
    while time.time() - start < duration:
        sum([i**2 for i in range(10000)])
    return {"status": "load simulated"}

Step 1.4: Test Locally

uvicorn app:app --reload

Visit http://127.0.0.1:8000/health and http://127.0.0.1:8000/metrics to verify.

Step 1.5: Create Unit Tests

Create tests/test_health.py:

from fastapi.testclient import TestClient
from app import app

client = TestClient(app)

def test_health():
    response = client.get("/health")
    assert response.status_code == 200
    assert response.json() == {"status": "healthy"}

Create tests/test_metrics.py:

from fastapi.testclient import TestClient
from app import app

client = TestClient(app)

def test_metrics():
    response = client.get("/metrics")
    assert response.status_code == 200
    data = response.json()
    assert "cpu" in data
    assert "memory" in data

Run tests:

pytest tests/

Step 1.6: Generate Requirements

pip freeze > requirements.txt

---

Phase 2: Containerization with Docker

Step 2.1: Create Dockerfile

Create Dockerfile:

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

EXPOSE 8000

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

Step 2.2: Build Docker Image

docker build -t health-metrics-service:1.0 .

Step 2.3: Run Container

docker run -d -p 8000:8000 --name health-metrics health-metrics-service:1.0

Step 2.4: Verify Container

docker ps
curl http://localhost:8000/health
docker logs health-metrics

---

Phase 3: Version Control & CI Setup

Step 3.1: Initialize Git Repository

git init
git add .
git commit -m "Initial commit: FastAPI health & metrics microservice"

Step 3.2: Create GitHub Repository

Create a new repository on GitHub named cloud-native-microservice-pipeline-monitor

git remote add origin https://github.com/<your-username>/cloud-native-microservice-pipeline-monitor.git
git branch -M main
git push -u origin main

Step 3.3: Setup GitHub Actions CI

Create .github/workflows/ci.yml:

name: CI - FastAPI Tests

on:
  push:
    branches: main
  pull_request:
    branches: main

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: |
          pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run unit tests
        run: 
          PYTHONPATH=. pytest

Step 3.4: Add Docker Build to CI

Update .github/workflows/ci.yml:

name: CI - Test & Docker Build

on:
  push:
    branches: main
  pull_request:
    branches: main

jobs:
  test-and-build:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: |
          pip install --upgrade pip
          pip install -r requirements.txt

      - name: Run unit tests
        run: 
          PYTHONPATH=. pytest

      - name: Build Docker image
        run: docker build -t health-metrics-service:ci .

---

Phase 4: AWS Infrastructure Setup

Step 4.1: Create ECR Repository

1. Navigate to AWS Console → ECR → Repositories

2. Click "Create repository"

3. Repository name: health-metrics-service

4. Region: us-east-1 (or your preferred region)

5. Note the repository URI

Step 4.2: Create IAM User for GitHub Actions

1. AWS Console → IAM → Users → Create user

2. Username: github-actions-ecr

3. Access type: Programmatic access

4. Attach policy: AmazonEC2ContainerRegistryPowerUser

5. Save Access Key ID and Secret Access Key

Step 4.3: Configure GitHub Secrets

Go to GitHub repo → Settings → Secrets and variables → Actions

Add these secrets:

• AWS_ACCESS_KEY_ID

• AWS_SECRET_ACCESS_KEY

• AWS_REGION (e.g., us-east-1)

• ECR_REPOSITORY (e.g., health-metrics-service)

• AWS_ACCOUNT_ID

Step 4.4: Update CI/CD to Push to ECR

Update .github/workflows/ci.yml:

name: CI - Test, Build & Push to ECR

on:
  push:
    branches: main
  pull_request:
    branches: main

jobs:
  build-and-push:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run tests
        run: pytest

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}

      - name: Login to Amazon ECR
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build and push Docker image
        env:
          ECR_REGISTRY: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com
          ECR_REPOSITORY: ${{ secrets.ECR_REPOSITORY }}
          IMAGE_TAG: latest
        run: |
          docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
          docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG

Step 4.5: Create EKS Cluster

1. AWS Console → EKS → Clusters → Create cluster

2. Cluster name: health-metrics-cluster

3. Kubernetes version: Latest stable

4. Configure networking (VPC, subnets)

5. Wait ~10 minutes for cluster creation

Step 4.6: Configure kubectl Access

aws eks update-kubeconfig --name health-metrics-cluster --region us-east-1

Step 4.7: Create EKS Access Entry for GitHub Actions

# Create access entry
aws eks create-access-entry \
  --cluster-name health-metrics-cluster \
  --region us-east-1 \
  --principal-arn arn:aws:iam::<AWS_ACCOUNT_ID>:user/github-actions-ecr \
  --type STANDARD

# Grant admin permissions
aws eks associate-access-policy \
  --cluster-name health-metrics-cluster \
  --region us-east-1 \
  --principal-arn arn:aws:iam::<AWS_ACCOUNT_ID>:user/github-actions-ecr \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope type=cluster

---

Phase 5: Kubernetes Deployment

Step 5.1: Create Kubernetes Manifests

Create k8s/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: health-metrics-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: health-metrics
  template:
    metadata:
      labels:
        app: health-metrics
    spec:
      containers:
      - name: health-metrics-container
        image: <AWS_ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com/health-metrics-service:latest
        ports:
        - containerPort: 8000

Create k8s/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: health-metrics-service
spec:
  type: LoadBalancer
  selector:
    app: health-metrics
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8000

Step 5.2: Add EKS Deployment to CI/CD

Add to .github/workflows/ci.yml:

      - name: Update kubeconfig for EKS
        run: |
          aws eks update-kubeconfig \
            --region ${{ secrets.AWS_REGION }} \
            --name health-metrics-cluster

      - name: Deploy to EKS
        run: kubectl apply -f k8s/

Step 5.3: Add GitHub Secret for EKS

Add EKS_CLUSTER_NAME: health-metrics-cluster to GitHub Secrets

---

Phase 6: Production Reliability Features

Step 6.1: Add Health Probes

Update k8s/deployment.yaml container spec:

        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 10
          periodSeconds: 10

        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 5
          periodSeconds: 10

Step 6.2: Configure Resource Limits

Add to container spec in k8s/deployment.yaml:

        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"

Step 6.3: Setup Horizontal Pod Autoscaler

Create k8s/hpa.yaml:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: health-metrics-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: health-metrics-deployment
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Apply:

kubectl apply -f k8s/hpa.yaml

Step 6.4: Configure Rolling Updates

Add to k8s/deployment.yaml spec:

  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

---

Phase 7: Observability & Monitoring

Step 7.1: Implement Prometheus Metrics

Update requirements.txt:

fastapi
uvicorn
prometheus-client
pytest

Update app.py:

from fastapi import FastAPI, Response
from prometheus_client import Counter, Gauge, generate_latest
import random, time

app = FastAPI()

# Prometheus metrics
REQUEST_COUNT = Counter('app_requests_total', 'Total API requests')
CPU_USAGE = Gauge('cpu_usage_percent', 'CPU usage percent')
MEMORY_USAGE = Gauge('memory_usage_percent', 'Memory usage percent')

@app.get("/health")
def health_check():
    return {"status": "healthy"}

@app.get("/metrics")
def metrics_endpoint():
    CPU_USAGE.set(random.uniform(0, 100))
    MEMORY_USAGE.set(random.uniform(0, 100))
    REQUEST_COUNT.inc()
    return Response(generate_latest(), media_type="text/plain")

@app.post("/simulate_load")
def simulate_load(duration: int = 5):
    start = time.time()
    while time.time() - start < duration:
        sum([i**2 for i in range(10000)])
    return {"status": "load simulated"}

Step 7.2: Update Test for Prometheus Format

Update tests/test_metrics.py:

from fastapi.testclient import TestClient
from app import app

client = TestClient(app)

def test_metrics():
    response = client.get("/metrics")
    assert response.status_code == 200
    content = response.text
    assert "cpu_usage_percent" in content
    assert "memory_usage_percent" in content

---

Phase 8: Multi-Environment Deployment

Step 8.1: Update CI/CD for Multiple Namespaces

Update .github/workflows/ci.yml:

      - name: Set namespace based on branch
        run: |
          if [ "${GITHUB_REF_NAME}" == "development" ]; then
            NAMESPACE="dev"
          elif [ "${GITHUB_REF_NAME}" == "staging" ]; then
            NAMESPACE="staging"
          elif [ "${GITHUB_REF_NAME}" == "main" ]; then
            NAMESPACE="prod"
          else
            echo "Unknown branch, skipping deployment"
            exit 0
          fi
          echo "Namespace=$NAMESPACE" >> $GITHUB_ENV

      - name: Create namespace if missing
        run: |
          kubectl get namespace ${{ env.Namespace }} || kubectl create namespace ${{ env.Namespace }}

      - name: Deploy to EKS
        run: |
          echo "Deploying to ${{ env.Namespace }} namespace..."
          kubectl apply -f k8s/ -n ${{ env.Namespace }}
          kubectl get all -n ${{ env.Namespace }}

---

Phase 9: Deployment Notifications

Step 9.1: Setup n8n Webhook

1. Create n8n workflow with Webhook node

2. Add Slack/Email notification node

3. Configure message template

4. Note webhook URL

Step 9.2: Add Notification Step to CI/CD

Add to .github/workflows/ci.yml:

      - name: Notify n8n
        if: always()
        run: |
          STATUS="success"
          if [ "${{ job.status }}" != "success" ]; then
            STATUS="failure"
          fi

          curl -X POST https://<n8n-webhook-url>/deployments \
            -H 'Content-Type: application/json' \
            -d '{
                  "service": "health-metrics-service",
                  "namespace": "${{ env.Namespace }}",
                  "status": "'"$STATUS"'"
                }'

---

Common Errors & Solutions

Error 1: EKS Authentication Failure

Symptoms:

error: You must be logged in to the server (Unauthorized)
couldn't get current server API group list

Root Cause: Missing EKS access entry for GitHub Actions IAM user

Solution:

# Create access entry
aws eks create-access-entry \
  --cluster-name health-metrics-cluster \
  --region us-east-1 \
  --principal-arn arn:aws:iam::<ACCOUNT_ID>:user/github-actions-ecr \
  --type STANDARD

# Grant permissions
aws eks associate-access-policy \
  --cluster-name health-metrics-cluster \
  --region us-east-1 \
  --principal-arn arn:aws:iam::<ACCOUNT_ID>:user/github-actions-ecr \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope type=cluster

Error 2: Stale Kubeconfig

Symptoms:

Unable to connect to the server: dial tcp: lookup <endpoint> on 127.0.0.53:53: no such host

Solution:

rm -f ~/.kube/config
aws eks update-kubeconfig --name health-metrics-cluster --region us-east-1
kubectl get nodes

Error 3: LoadBalancer Pending

Symptoms:

EXTERNAL-IP: <pending>

Root Cause: AWS Load Balancer Controller not installed

Quick Fix - Use NodePort:

# Switch to NodePort
kubectl patch service health-metrics-service -n prod -p '{"spec":{"type":"NodePort"}}'

# Get NodePort
kubectl get service health-metrics-service -n prod

# Open security group
aws ec2 authorize-security-group-ingress \
  --group-id <node-sg> \
  --protocol tcp \
  --port <node-port> \
  --cidr 0.0.0.0/0

Proper Solution - Install AWS Load Balancer Controller:

# Download IAM policy
curl -o iam_policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.7.1/docs/install/iam_policy.json

# Create policy
aws iam create-policy \
  --policy-name AWSLoadBalancerControllerIAMPolicy \
  --policy-document file://iam_policy.json

# Associate OIDC provider
eksctl utils associate-iam-oidc-provider \
  --region us-east-1 \
  --cluster health-metrics-cluster \
  --approve

# Create service account
eksctl create iamserviceaccount \
  --cluster=health-metrics-cluster \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --role-name AmazonEKSLoadBalancerControllerRole \
  --attach-policy-arn=arn:aws:iam::<ACCOUNT_ID>:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve \
  --region=us-east-1

# Install controller via Helm
helm repo add eks https://aws.github.io/eks-charts
helm repo update

VPC_ID=$(aws eks describe-cluster \
  --name health-metrics-cluster \
  --region us-east-1 \
  --query "cluster.resourcesVpcConfig.vpcId" \
  --output text)

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=health-metrics-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set region=us-east-1 \
  --set vpcId=$VPC_ID

Error 4: JSONDecodeError in Tests

Symptoms:

json.decoder.JSONDecodeError: Expecting value: line 1 column 1

Root Cause: Prometheus metrics return text format, not JSON

Solution:

Update tests/test_metrics.py:

def test_metrics():
    response = client.get("/metrics")
    assert response.status_code == 200
    content = response.text
    assert "cpu_usage_percent" in content
    assert "memory_usage_percent" in content

Error 5: OIDC Provider Missing

Symptoms:

Error: no IAM OIDC provider associated with cluster

Solution:

eksctl utils associate-iam-oidc-provider \
  --region us-east-1 \
  --cluster health-metrics-cluster \
  --approve

Error 6: pytest Cannot Find Module (ModuleNotFoundError)

Symptoms:

ModuleNotFoundError: No module named 'app'
ImportError: cannot import name 'app' from 'app'

Root Cause: Python cannot locate your module because the project root isn't in the PYTHONPATH

Solutions:

Option 1: Quick Fix (Temporary)

PYTHONPATH=. pytest tests/

Option 2: Set PYTHONPATH in Virtual Environment (Persistent)

# Add to your virtual environment activation
echo 'export PYTHONPATH="${PYTHONPATH}:$(pwd)"' >> venv/bin/activate

# Reactivate virtual environment
deactivate
source venv/bin/activate

# Now pytest works normally
pytest tests/

Option 3: Create pytest Configuration File (Recommended)

Create pytest.ini in project root:

[pytest]
pythonpath = .
testpaths = tests

Or create pyproject.toml:

[tool.pytest.ini_options]
pythonpath = ["."]
testpaths = ["tests"]

Conclusion

Developer Push to GitHub
         ↓
GitHub Actions CI/CD
         ↓
Run Tests (pytest)
         ↓
Build Docker Image
         ↓
Push to AWS ECR
         ↓
Deploy to EKS (kubectl)
         ↓
Kubernetes Cluster
  ├─ Deployment (2-5 replicas)
  ├─ HPA (auto-scaling)
  ├─ Service (LoadBalancer)
  └─ Health Probes
         ↓
n8n Notification (Slack/Email)

Resources

• FastAPI Documentation

• Kubernetes Documentation

• AWS EKS Best Practices

• GitHub Actions Documentation

• Prometheus Documentation