# Building a Production-Ready Cloud-Native Microservice with Complete CI/CD Pipeline on AWS EKS

### 📋 Project Overview

This comprehensive guide walks you through building a **production-grade cloud-native microservice** using FastAPI, Docker, Kubernetes (AWS EKS), and GitHub Actions CI/CD. You'll learn how to implement enterprise-level DevOps practices including automated testing, container orchestration, auto-scaling, health monitoring, and deployment notifications.

**What You'll Build:**

* Automated CI/CD pipeline using GitHub Actions
    
* Kubernetes deployment on AWS EKS with auto-scaling
    
* Multi-environment setup (dev, staging, production)
    
* Production observability and notifications
    

**Tech Stack:** Python, FastAPI, Docker, Kubernetes, AWS EKS, ECR, GitHub Actions, Prometheus, n8n

**GitHub Repository:** `https://github.com/sachindumalshan/cloud-native-microservice-pipeline-monitor`

---

## 📑 Table of Contents

1. [Prerequisites](#prerequisites)
    
2. [Phase 1: Local Development Setup](#phase-1-local-development-setup)
    
3. [Phase 2: Containerization with Docker](#phase-2-containerization-with-docker)
    
4. [Phase 3: Version Control & CI Setup](#phase-3-version-control--ci-setup)
    
5. [Phase 4: AWS Infrastructure Setup](#phase-4-aws-infrastructure-setup)
    
6. [Phase 5: Kubernetes Deployment](#phase-5-kubernetes-deployment)
    
7. [Phase 6: Production Reliability Features](#phase-6-production-reliability-features)
    
8. [Phase 7: Observability & Monitoring](#phase-7-observability--monitoring)
    
9. [Phase 8: Multi-Environment Deployment](#phase-8-multi-environment-deployment)
    
10. [Phase 9: Deployment Notifications](#phase-9-deployment-notifications)
    
11. [Common Errors & Solutions](#common-errors--solutions)
    
12. [Conclusion & Key Takeaways](#conclusion--key-takeaways)
    

---

## Prerequisites

Before starting, ensure you have:

* Python 3.12+ installed
    
* Docker Desktop running
    
* AWS Account with appropriate permissions
    
* GitHub account
    
* Basic knowledge of Python, REST APIs, and command line
    
* kubectl and AWS CLI installed
    

---

## Phase 1: Local Development Setup

### Step 1.1: Create Project Structure

```bash
mkdir health_metrics_service
cd health_metrics_service
python3 -m venv venv
source venv/bin/activate
```

### Step 1.2: Install Dependencies

```bash
pip install fastapi uvicorn pytest
```

### Step 1.3: Create FastAPI Application

Create `app.py`:

```python
from fastapi import FastAPI
import random, time

app = FastAPI()

@app.get("/health")
def health_check():
    return {"status": "healthy"}

@app.get("/metrics")
def metrics():
    cpu_load = random.uniform(0, 100)
    memory_usage = random.uniform(0, 100)
    return {"cpu": cpu_load, "memory": memory_usage}

@app.post("/simulate_load")
def simulate_load(duration: int = 5):
    start = time.time()
    while time.time() - start < duration:
        sum([i**2 for i in range(10000)])
    return {"status": "load simulated"}
```

### Step 1.4: Test Locally

```bash
uvicorn app:app --reload
```

Visit `http://127.0.0.1:8000/health` and `http://127.0.0.1:8000/metrics` to verify.

### Step 1.5: Create Unit Tests

Create `tests/test_health.py`:

```python
from fastapi.testclient import TestClient
from app import app

client = TestClient(app)

def test_health():
    response = client.get("/health")
    assert response.status_code == 200
    assert response.json() == {"status": "healthy"}
```

Create `tests/test_metrics.py`:

```python
from fastapi.testclient import TestClient
from app import app

client = TestClient(app)

def test_metrics():
    response = client.get("/metrics")
    assert response.status_code == 200
    data = response.json()
    assert "cpu" in data
    assert "memory" in data
```

Run tests:

```bash
pytest tests/
```

### Step 1.6: Generate Requirements

```bash
pip freeze > requirements.txt
```

---

## Phase 2: Containerization with Docker

### Step 2.1: Create Dockerfile

Create `Dockerfile`:

```dockerfile
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

EXPOSE 8000

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
```

### Step 2.2: Build Docker Image

```bash
docker build -t health-metrics-service:1.0 .
```

### Step 2.3: Run Container

```bash
docker run -d -p 8000:8000 --name health-metrics health-metrics-service:1.0
```

### Step 2.4: Verify Container

```bash
docker ps
curl http://localhost:8000/health
docker logs health-metrics
```

---

## Phase 3: Version Control & CI Setup

### Step 3.1: Initialize Git Repository

```bash
git init
git add .
git commit -m "Initial commit: FastAPI health & metrics microservice"
```

### Step 3.2: Create GitHub Repository

Create a new repository on GitHub named `cloud-native-microservice-pipeline-monitor`

```bash
git remote add origin https://github.com/<your-username>/cloud-native-microservice-pipeline-monitor.git
git branch -M main
git push -u origin main
```

### Step 3.3: Setup GitHub Actions CI

Create `.github/workflows/ci.yml`:

```yaml
name: CI - FastAPI Tests

on:
  push:
    branches: main
  pull_request:
    branches: main

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      
      - name: Install dependencies
        run: |
          pip install --upgrade pip
          pip install -r requirements.txt
      
      - name: Run unit tests
        run: 
          PYTHONPATH=. pytest
```

### Step 3.4: Add Docker Build to CI

Update `.github/workflows/ci.yml`:

```yaml
name: CI - Test & Docker Build

on:
  push:
    branches: main
  pull_request:
    branches: main

jobs:
  test-and-build:
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      
      - name: Install dependencies
        run: |
          pip install --upgrade pip
          pip install -r requirements.txt
      
      - name: Run unit tests
        run: 
          PYTHONPATH=. pytest
      
      - name: Build Docker image
        run: docker build -t health-metrics-service:ci .
```

---

## Phase 4: AWS Infrastructure Setup

### Step 4.1: Create ECR Repository

1. Navigate to AWS Console → ECR → Repositories
    
2. Click "Create repository"
    
3. Repository name: `health-metrics-service`
    
4. Region: `us-east-1` (or your preferred region)
    
5. Note the repository URI
    

### Step 4.2: Create IAM User for GitHub Actions

1. AWS Console → IAM → Users → Create user
    
2. Username: `github-actions-ecr`
    
3. Access type: Programmatic access
    
4. Attach policy: `AmazonEC2ContainerRegistryPowerUser`
    
5. Save Access Key ID and Secret Access Key
    

### Step 4.3: Configure GitHub Secrets

Go to GitHub repo → Settings → Secrets and variables → Actions

Add these secrets:

* `AWS_ACCESS_KEY_ID`
    
* `AWS_SECRET_ACCESS_KEY`
    
* `AWS_REGION` (e.g., `us-east-1`)
    
* `ECR_REPOSITORY` (e.g., `health-metrics-service`)
    
* `AWS_ACCOUNT_ID`
    

### Step 4.4: Update CI/CD to Push to ECR

Update `.github/workflows/ci.yml`:

```yaml
name: CI - Test, Build & Push to ECR

on:
  push:
    branches: main
  pull_request:
    branches: main

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      
      - name: Install dependencies
        run: pip install -r requirements.txt
      
      - name: Run tests
        run: pytest
      
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}
      
      - name: Login to Amazon ECR
        uses: aws-actions/amazon-ecr-login@v2
      
      - name: Build and push Docker image
        env:
          ECR_REGISTRY: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.${{ secrets.AWS_REGION }}.amazonaws.com
          ECR_REPOSITORY: ${{ secrets.ECR_REPOSITORY }}
          IMAGE_TAG: latest
        run: |
          docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
          docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
```

### Step 4.5: Create EKS Cluster

1. AWS Console → EKS → Clusters → Create cluster
    
2. Cluster name: `health-metrics-cluster`
    
3. Kubernetes version: Latest stable
    
4. Configure networking (VPC, subnets)
    
5. Wait ~10 minutes for cluster creation
    

### Step 4.6: Configure kubectl Access

```bash
aws eks update-kubeconfig --name health-metrics-cluster --region us-east-1
```

### Step 4.7: Create EKS Access Entry for GitHub Actions

```bash
# Create access entry
aws eks create-access-entry \
  --cluster-name health-metrics-cluster \
  --region us-east-1 \
  --principal-arn arn:aws:iam::<AWS_ACCOUNT_ID>:user/github-actions-ecr \
  --type STANDARD

# Grant admin permissions
aws eks associate-access-policy \
  --cluster-name health-metrics-cluster \
  --region us-east-1 \
  --principal-arn arn:aws:iam::<AWS_ACCOUNT_ID>:user/github-actions-ecr \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope type=cluster
```

---

## Phase 5: Kubernetes Deployment

### Step 5.1: Create Kubernetes Manifests

Create `k8s/deployment.yaml`:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: health-metrics-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: health-metrics
  template:
    metadata:
      labels:
        app: health-metrics
    spec:
      containers:
      - name: health-metrics-container
        image: <AWS_ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com/health-metrics-service:latest
        ports:
        - containerPort: 8000
```

Create `k8s/service.yaml`:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: health-metrics-service
spec:
  type: LoadBalancer
  selector:
    app: health-metrics
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8000
```

### Step 5.2: Add EKS Deployment to CI/CD

Add to `.github/workflows/ci.yml`:

```yaml
      - name: Update kubeconfig for EKS
        run: |
          aws eks update-kubeconfig \
            --region ${{ secrets.AWS_REGION }} \
            --name health-metrics-cluster
      
      - name: Deploy to EKS
        run: kubectl apply -f k8s/
```

### Step 5.3: Add GitHub Secret for EKS

Add `EKS_CLUSTER_NAME: health-metrics-cluster` to GitHub Secrets

---

## Phase 6: Production Reliability Features

### Step 6.1: Add Health Probes

Update `k8s/deployment.yaml` container spec:

```yaml
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 10
          periodSeconds: 10
        
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 5
          periodSeconds: 10
```

### Step 6.2: Configure Resource Limits

Add to container spec in `k8s/deployment.yaml`:

```yaml
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"
```

### Step 6.3: Setup Horizontal Pod Autoscaler

Create `k8s/hpa.yaml`:

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: health-metrics-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: health-metrics-deployment
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
```

Apply:

```bash
kubectl apply -f k8s/hpa.yaml
```

### Step 6.4: Configure Rolling Updates

Add to `k8s/deployment.yaml` spec:

```yaml
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
```

---

## Phase 7: Observability & Monitoring

### Step 7.1: Implement Prometheus Metrics

Update `requirements.txt`:

```bash
fastapi
uvicorn
prometheus-client
pytest
```

Update `app.py`:

```python
from fastapi import FastAPI, Response
from prometheus_client import Counter, Gauge, generate_latest
import random, time

app = FastAPI()

# Prometheus metrics
REQUEST_COUNT = Counter('app_requests_total', 'Total API requests')
CPU_USAGE = Gauge('cpu_usage_percent', 'CPU usage percent')
MEMORY_USAGE = Gauge('memory_usage_percent', 'Memory usage percent')

@app.get("/health")
def health_check():
    return {"status": "healthy"}

@app.get("/metrics")
def metrics_endpoint():
    CPU_USAGE.set(random.uniform(0, 100))
    MEMORY_USAGE.set(random.uniform(0, 100))
    REQUEST_COUNT.inc()
    return Response(generate_latest(), media_type="text/plain")

@app.post("/simulate_load")
def simulate_load(duration: int = 5):
    start = time.time()
    while time.time() - start < duration:
        sum([i**2 for i in range(10000)])
    return {"status": "load simulated"}
```

### Step 7.2: Update Test for Prometheus Format

Update `tests/test_metrics.py`:

```python
from fastapi.testclient import TestClient
from app import app

client = TestClient(app)

def test_metrics():
    response = client.get("/metrics")
    assert response.status_code == 200
    content = response.text
    assert "cpu_usage_percent" in content
    assert "memory_usage_percent" in content
```

---

## Phase 8: Multi-Environment Deployment

### Step 8.1: Update CI/CD for Multiple Namespaces

Update `.github/workflows/ci.yml`:

```yaml
      - name: Set namespace based on branch
        run: |
          if [ "${GITHUB_REF_NAME}" == "development" ]; then
            NAMESPACE="dev"
          elif [ "${GITHUB_REF_NAME}" == "staging" ]; then
            NAMESPACE="staging"
          elif [ "${GITHUB_REF_NAME}" == "main" ]; then
            NAMESPACE="prod"
          else
            echo "Unknown branch, skipping deployment"
            exit 0
          fi
          echo "Namespace=$NAMESPACE" >> $GITHUB_ENV
      
      - name: Create namespace if missing
        run: |
          kubectl get namespace ${{ env.Namespace }} || kubectl create namespace ${{ env.Namespace }}
      
      - name: Deploy to EKS
        run: |
          echo "Deploying to ${{ env.Namespace }} namespace..."
          kubectl apply -f k8s/ -n ${{ env.Namespace }}
          kubectl get all -n ${{ env.Namespace }}
```

---

## Phase 9: Deployment Notifications

### Step 9.1: Setup n8n Webhook

1. Create n8n workflow with Webhook node
    
2. Add Slack/Email notification node
    
3. Configure message template
    
4. Note webhook URL
    

### Step 9.2: Add Notification Step to CI/CD

Add to `.github/workflows/ci.yml`:

```yaml
      - name: Notify n8n
        if: always()
        run: |
          STATUS="success"
          if [ "${{ job.status }}" != "success" ]; then
            STATUS="failure"
          fi
          
          curl -X POST https://<n8n-webhook-url>/deployments \
            -H 'Content-Type: application/json' \
            -d '{
                  "service": "health-metrics-service",
                  "namespace": "${{ env.Namespace }}",
                  "status": "'"$STATUS"'"
                }'
```

---

## Common Errors & Solutions

### Error 1: EKS Authentication Failure

**Symptoms:**

```bash
error: You must be logged in to the server (Unauthorized)
couldn't get current server API group list
```

**Root Cause:** Missing EKS access entry for GitHub Actions IAM user

**Solution:**

```bash
# Create access entry
aws eks create-access-entry \
  --cluster-name health-metrics-cluster \
  --region us-east-1 \
  --principal-arn arn:aws:iam::<ACCOUNT_ID>:user/github-actions-ecr \
  --type STANDARD

# Grant permissions
aws eks associate-access-policy \
  --cluster-name health-metrics-cluster \
  --region us-east-1 \
  --principal-arn arn:aws:iam::<ACCOUNT_ID>:user/github-actions-ecr \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope type=cluster
```

### Error 2: Stale Kubeconfig

**Symptoms:**

```bash
Unable to connect to the server: dial tcp: lookup <endpoint> on 127.0.0.53:53: no such host
```

**Solution:**

```bash
rm -f ~/.kube/config
aws eks update-kubeconfig --name health-metrics-cluster --region us-east-1
kubectl get nodes
```

### Error 3: LoadBalancer Pending

**Symptoms:**

```bash
EXTERNAL-IP: <pending>
```

**Root Cause:** AWS Load Balancer Controller not installed

**Quick Fix - Use NodePort:**

```bash
# Switch to NodePort
kubectl patch service health-metrics-service -n prod -p '{"spec":{"type":"NodePort"}}'

# Get NodePort
kubectl get service health-metrics-service -n prod

# Open security group
aws ec2 authorize-security-group-ingress \
  --group-id <node-sg> \
  --protocol tcp \
  --port <node-port> \
  --cidr 0.0.0.0/0
```

**Proper Solution - Install AWS Load Balancer Controller:**

```bash
# Download IAM policy
curl -o iam_policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.7.1/docs/install/iam_policy.json

# Create policy
aws iam create-policy \
  --policy-name AWSLoadBalancerControllerIAMPolicy \
  --policy-document file://iam_policy.json

# Associate OIDC provider
eksctl utils associate-iam-oidc-provider \
  --region us-east-1 \
  --cluster health-metrics-cluster \
  --approve

# Create service account
eksctl create iamserviceaccount \
  --cluster=health-metrics-cluster \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --role-name AmazonEKSLoadBalancerControllerRole \
  --attach-policy-arn=arn:aws:iam::<ACCOUNT_ID>:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve \
  --region=us-east-1

# Install controller via Helm
helm repo add eks https://aws.github.io/eks-charts
helm repo update

VPC_ID=$(aws eks describe-cluster \
  --name health-metrics-cluster \
  --region us-east-1 \
  --query "cluster.resourcesVpcConfig.vpcId" \
  --output text)

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=health-metrics-cluster \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set region=us-east-1 \
  --set vpcId=$VPC_ID
```

### Error 4: JSONDecodeError in Tests

**Symptoms:**

```bash
json.decoder.JSONDecodeError: Expecting value: line 1 column 1
```

**Root Cause:** Prometheus metrics return text format, not JSON

**Solution:**

Update `tests/test_metrics.py`:

```python
def test_metrics():
    response = client.get("/metrics")
    assert response.status_code == 200
    content = response.text
    assert "cpu_usage_percent" in content
    assert "memory_usage_percent" in content
```

### Error 5: OIDC Provider Missing

**Symptoms:**

```bash
Error: no IAM OIDC provider associated with cluster
```

**Solution:**

```bash
eksctl utils associate-iam-oidc-provider \
  --region us-east-1 \
  --cluster health-metrics-cluster \
  --approve
```

### Error 6: pytest Cannot Find Module (ModuleNotFoundError)

**Symptoms:**

```bash
ModuleNotFoundError: No module named 'app'
ImportError: cannot import name 'app' from 'app'
```

**Root Cause:** Python cannot locate your module because the project root isn't in the PYTHONPATH

**Solutions:**

**Option 1: Quick Fix (Temporary)**

```bash
PYTHONPATH=. pytest tests/
```

**Option 2: Set PYTHONPATH in Virtual Environment (Persistent)**

```bash
# Add to your virtual environment activation
echo 'export PYTHONPATH="${PYTHONPATH}:$(pwd)"' >> venv/bin/activate

# Reactivate virtual environment
deactivate
source venv/bin/activate

# Now pytest works normally
pytest tests/
```

**Option 3: Create pytest Configuration File (Recommended)**

Create `pytest.ini` in project root:

```ini
[pytest]
pythonpath = .
testpaths = tests
```

Or create `pyproject.toml`:

```toml
[tool.pytest.ini_options]
pythonpath = ["."]
testpaths = ["tests"]
```

## Conclusion

```bash
Developer Push to GitHub
         ↓
GitHub Actions CI/CD
         ↓
Run Tests (pytest)
         ↓
Build Docker Image
         ↓
Push to AWS ECR
         ↓
Deploy to EKS (kubectl)
         ↓
Kubernetes Cluster
  ├─ Deployment (2-5 replicas)
  ├─ HPA (auto-scaling)
  ├─ Service (LoadBalancer)
  └─ Health Probes
         ↓
n8n Notification (Slack/Email)
```

## Resources

* [FastAPI Documentation](https://fastapi.tiangolo.com/)
    
* [Kubernetes Documentation](https://kubernetes.io/docs/)
    
* [AWS EKS Best Practices](https://aws.github.io/aws-eks-best-practices/)
    
* [GitHub Actions Documentation](https://docs.github.com/en/actions)
    
* [Prometheus Documentation](https://prometheus.io/docs/)