# Deploying a Full-Stack Application Across Hybrid Cloud Infrastructure

This guide covers deploying a simple full‑stack app with an AWS frontend and backend that talks to an on‑prem database running in a Proxmox LXC container, plus a basic monitoring stack. It assumes you’ve completed the network setup in Article 1 and can reach ct-db from EC2 over Tailscale.

## Table of contents

* Overview and architecture
    
* Create AWS VPC (public + private subnets)
    
* Provision EC2 instances (frontend public, backend private)
    
* Configure frontend → backend communication (Nginx reverse proxy)
    
* Create two LXC in Proxmox (ct-db, ct-monitor)
    
* Proxmox host ↔ CT connectivity checks
    
* Backend EC2 ↔ ct-db over Tailscale
    
* Backend (Node.js + MariaDB client) setup
    
* Database on ct-db (MariaDB recommended)
    
* Monitoring on ct-monitor (Prometheus + Grafana)
    
* End-to-end test and verification
    
* Security hardening
    

---

## Overview and architecture

High-level flow: A public frontend EC2 serves the website and proxies API requests to a private backend EC2. The backend talks to an on‑prem database (ct-db) over a Tailscale mesh to your Proxmox host. Monitoring runs in another LXC (ct-monitor).

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1762714670101/2838acb6-e061-4089-aaa0-4140f468dec0.jpeg align="center")

Prereqs from Article 1:

* Proxmox with two LXC containers
    
    * ct-db: 192.168.8.101 (MariaDB/PostgreSQL)
        
    * ct-monitor: 192.168.8.102 (Prometheus + Grafana)
        
* Tailscale configured
    
    * Proxmox advertises 192.168.8.0/24
        
    * EC2 instances accept routes
        
* Verified EC2 → ct-db connectivity (ping + TCP/3306)
    

---

## Create AWS VPC (public + private subnets)

Minimal setup:

1. VPC with one public and one private subnet
    
    * Public subnet: Internet Gateway + route for 0.0.0.0/0
        
    * Private subnet: NAT Gateway + route for 0.0.0.0/0 via NAT
        
2. Route tables
    
    * Associate the public subnet with the IGW route table
        
    * Associate the private subnet with the NAT route table
        

Keep the backend private; you’ll reach it from the frontend using its private IP and an Nginx reverse proxy.

---

## Provision EC2 instances (frontend public, backend private)

Create two EC2 instances in your VPC: one in the public subnet for the frontend and one in the private subnet for the backend.

### Frontend EC2 (Public Subnet)

| Configuration | Value |
| --- | --- |
| AMI | Amazon Linux 2023 or Ubuntu 22.04 LTS |
| Instance Type | t2.micro or t2.nano |
| Subnet | Public subnet |
| Auto-assign Public IP | Enable |
| Security Group | SG-frontend |

**SG-frontend Rules:**

* Port 22 (SSH) from your IP
    
* Port 80 (HTTP) from 0.0.0.0/0
    
* Port 443 (HTTPS) from 0.0.0.0/0
    

### Backend EC2 (Private Subnet)

| Configuration | Value |
| --- | --- |
| AMI | Amazon Linux 2023 or Ubuntu 22.04 LTS |
| Instance Type | t2.micro or t2.nano |
| Subnet | Private subnet |
| Auto-assign Public IP | Disable |
| Security Group | SG-backend |

**SG-backend Rules:**

* Port 22 (SSH) from SG-frontend
    
* Port 3000 (TCP) from SG-frontend
    

---

## SSH Access and Bastion Setup

### Connect to Frontend EC2

Log into the frontend VM using SSH with your PEM key:

```bash
ssh -i hybrid-cloud.pem ec2-user@<FRONTEND_PUBLIC_IP>
```

### Connect to Backend EC2 (via bastion)

The backend instance has no public IP, so use the frontend as a bastion host.

**Step 1:** Copy the PEM key to the frontend instance:

```bash
scp -i hybrid-cloud.pem hybrid-cloud.pem ec2-user@<FRONTEND_PUBLIC_IP>:/home/ec2-user/
```

**Step 2:** SSH from the frontend to the backend using its private IP:

```bash
ssh -i ~/hybrid-cloud.pem ec2-user@<BACKEND_PRIVATE_IP>
```

---

## Configure Frontend (Nginx + Static Site + Reverse Proxy)

This section covers installing Nginx, creating the static HTML pages, and configuring the reverse proxy to route API requests to the backend.

### Install and Enable Nginx

Update packages and install Nginx:

```bash
sudo yum update -y && sudo yum upgrade -y
sudo yum install -y nginx
sudo systemctl enable --now nginx
```

### Create Web Root and HTML Files

Create the web root directory:

```bash
sudo mkdir -p /var/www/html
cd /var/www/html
```

Create `index.html`:

```html
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Hybrid Cloud Platform</title>
  <style>
    * {
      margin: 0;
      padding: 0;
      box-sizing: border-box;
    }

    body {
      font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
      background: linear-gradient(135deg, #31694E 0%, #658C58 100%);
      min-height: 100vh;
      display: flex;
      justify-content: center;
      align-items: center;
      padding: 20px;
    }

    .container {
      background: #F0E491;
      border-radius: 20px;
      box-shadow: 0 20px 60px rgba(0, 0, 0, 0.3);
      padding: 60px 40px;
      max-width: 600px;
      width: 100%;
      text-align: center;
    }

    h1 {
      color: #31694E;
      font-size: 2.5em;
      margin-bottom: 15px;
      font-weight: 700;
    }

    .subtitle {
      color: #658C58;
      font-size: 1.2em;
      margin-bottom: 40px;
      font-weight: 500;
    }

    .project-scope {
      background: white;
      border-left: 5px solid #BBC863;
      padding: 20px;
      margin: 30px 0;
      border-radius: 8px;
      text-align: left;
    }

    .project-scope h2 {
      color: #31694E;
      font-size: 1.3em;
      margin-bottom: 10px;
    }

    .project-scope p {
      color: #333;
      line-height: 1.6;
      font-size: 1.05em;
    }

    .tech-badges {
      display: flex;
      justify-content: center;
      gap: 15px;
      margin: 20px 0;
      flex-wrap: wrap;
    }

    .badge {
      background: #658C58;
      color: #F0E491;
      padding: 8px 20px;
      border-radius: 20px;
      font-size: 0.9em;
      font-weight: 600;
    }

    .cta-button {
      display: inline-block;
      background: #31694E;
      color: #F0E491;
      text-decoration: none;
      padding: 18px 50px;
      border-radius: 50px;
      font-size: 1.3em;
      font-weight: 700;
      margin-top: 30px;
      transition: all 0.3s ease;
      box-shadow: 0 8px 20px rgba(49, 105, 78, 0.4);
      position: relative;
      overflow: hidden;
    }

    .cta-button::before {
      content: '';
      position: absolute;
      top: 0;
      left: -100%;
      width: 100%;
      height: 100%;
      background: linear-gradient(90deg, transparent, rgba(255, 255, 255, 0.2), transparent);
      transition: left 0.5s;
    }

    .cta-button:hover::before {
      left: 100%;
    }

    .cta-button:hover {
      background: #254d3a;
      transform: translateY(-3px);
      box-shadow: 0 12px 30px rgba(49, 105, 78, 0.5);
    }

    .highlight-box {
      background: #BBC863;
      color: #31694E;
      padding: 5px 15px;
      border-radius: 5px;
      display: inline-block;
      font-weight: 700;
      margin-top: 10px;
      font-size: 0.95em;
    }

    @media (max-width: 600px) {
      .container {
        padding: 40px 25px;
      }

      h1 {
        font-size: 2em;
      }

      .subtitle {
        font-size: 1em;
      }
    }
  </style>
</head>
<body>
  <div class="container">
    <h1>🚀 Hybrid Cloud Platform</h1>
    <p class="subtitle">Seamless Integration Across Environments</p>
    
    <div class="project-scope">
      <h2>Project Scope</h2>
      <p><strong>Hybrid Cloud Infrastructure:</strong> AWS Cloud + Proxmox Homelab</p>
      <div class="tech-badges">
        <span class="badge">AWS</span>
        <span class="badge">Proxmox</span>
        <span class="badge">Homelab</span>
      </div>
    </div>

    <a href="register.html" class="cta-button">Register as a User</a>
  </div>
</body>
</html>
```

Create `register.html`:

```xml
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Register - Hybrid Cloud Platform</title>
  <style>
    * {
      margin: 0;
      padding: 0;
      box-sizing: border-box;
    }

    body {
      font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
      background: linear-gradient(135deg, #31694E 0%, #658C58 100%);
      min-height: 100vh;
      display: flex;
      justify-content: center;
      align-items: center;
      padding: 20px;
    }

    .container {
      background: #F0E491;
      border-radius: 20px;
      box-shadow: 0 20px 60px rgba(0, 0, 0, 0.3);
      padding: 50px 40px;
      max-width: 500px;
      width: 100%;
    }

    .header {
      text-align: center;
      margin-bottom: 40px;
    }

    h2 {
      color: #31694E;
      font-size: 2.2em;
      margin-bottom: 10px;
      font-weight: 700;
    }

    .subtitle {
      color: #658C58;
      font-size: 1em;
      margin-bottom: 10px;
    }

    .back-link {
      display: inline-block;
      color: #31694E;
      text-decoration: none;
      font-size: 0.95em;
      margin-bottom: 20px;
      transition: color 0.3s;
    }

    .back-link:hover {
      color: #658C58;
    }

    .form-group {
      margin-bottom: 25px;
    }

    label {
      display: block;
      color: #31694E;
      font-weight: 600;
      margin-bottom: 8px;
      font-size: 0.95em;
    }

    input[type="text"],
    input[type="password"] {
      width: 100%;
      padding: 15px;
      border: 2px solid #BBC863;
      border-radius: 10px;
      font-size: 1em;
      background: white;
      color: #31694E;
      transition: all 0.3s;
      outline: none;
    }

    input[type="text"]:focus,
    input[type="password"]:focus {
      border-color: #658C58;
      box-shadow: 0 0 0 3px rgba(101, 140, 88, 0.1);
    }

    input::placeholder {
      color: #BBC863;
    }

    .btn-submit {
      width: 100%;
      padding: 16px;
      background: #31694E;
      color: #F0E491;
      border: none;
      border-radius: 10px;
      font-size: 1.1em;
      font-weight: 700;
      cursor: pointer;
      transition: all 0.3s;
      box-shadow: 0 6px 20px rgba(49, 105, 78, 0.3);
      margin-top: 10px;
    }

    .btn-submit:hover {
      background: #254d3a;
      transform: translateY(-2px);
      box-shadow: 0 8px 25px rgba(49, 105, 78, 0.4);
    }

    .btn-submit:active {
      transform: translateY(0);
    }

    .btn-submit:disabled {
      background: #BBC863;
      cursor: not-allowed;
      transform: none;
    }

    .info-box {
      background: white;
      border-left: 4px solid #BBC863;
      padding: 15px;
      border-radius: 8px;
      margin-top: 25px;
      font-size: 0.9em;
      color: #31694E;
    }

    .alert {
      padding: 15px;
      border-radius: 8px;
      margin-top: 20px;
      font-weight: 500;
      display: none;
    }

    .alert.success {
      background: #BBC863;
      color: #31694E;
      display: block;
    }

    .alert.error {
      background: #ff6b6b;
      color: white;
      display: block;
    }

    @media (max-width: 600px) {
      .container {
        padding: 35px 25px;
      }

      h2 {
        font-size: 1.8em;
      }
    }
  </style>
</head>
<body>
  <div class="container">
    <a href="index.html" class="back-link">← Back to Home</a>
    
    <div class="header">
      <h2>Create Account</h2>
      <p class="subtitle">Join the Hybrid Cloud Platform</p>
    </div>

    <form id="regForm">
      <div class="form-group">
        <label for="username">Username</label>
        <input type="text" id="username" placeholder="Enter your username" required>
      </div>

      <div class="form-group">
        <label for="password">Password</label>
        <input type="password" id="password" placeholder="Enter your password" required>
      </div>

      <button type="submit" class="btn-submit">Register Account</button>

      <div id="alertBox" class="alert"></div>
    </form>

    <div class="info-box">
      <strong>🔒 Secure Registration</strong><br>
      Your credentials will be securely stored in our hybrid cloud infrastructure.
    </div>
  </div>

  <script>
    document.getElementById('regForm').addEventListener('submit', async (e) => {
      e.preventDefault();
      
      const alertBox = document.getElementById('alertBox');
      const submitBtn = e.target.querySelector('.btn-submit');
      const username = document.getElementById('username').value;
      const password = document.getElementById('password').value;
      
      // Reset alert
      alertBox.className = 'alert';
      alertBox.textContent = '';
      
      // Disable button during request
      submitBtn.disabled = true;
      submitBtn.textContent = 'Registering...';
      
      try {
        const res = await fetch('/api/register', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify({ username, password })
        });
        
        // Check if response is ok
        if (!res.ok) {
          throw new Error(`Server returned ${res.status}: ${res.statusText}`);
        }
        
        // Check if response is JSON
        const contentType = res.headers.get('content-type');
        if (!contentType || !contentType.includes('application/json')) {
          const text = await res.text();
          throw new Error(`Expected JSON but got: ${text.substring(0, 100)}`);
        }
        
        const data = await res.json();
        
        if (data.success) {
          alertBox.className = 'alert success';
          alertBox.textContent = `✓ User ${data.user.username} registered successfully!`;
          document.getElementById('regForm').reset();
        } else {
          alertBox.className = 'alert error';
          alertBox.textContent = `✗ Error: ${data.message}`;
        }
      } catch (error) {
        alertBox.className = 'alert error';
        alertBox.textContent = `✗ Error: ${error.message}`;
        console.error('Registration error:', error);
      } finally {
        // Re-enable button
        submitBtn.disabled = false;
        submitBtn.textContent = 'Register Account';
      }
    });
  </script>
</body>
</html>
```

### Configure Nginx Reverse Proxy

Edit the Nginx configuration (Amazon Linux path: `/etc/nginx/nginx.conf` or create a file under `/etc/nginx/conf.d/`):

```nginx
server {
    listen 80;
    server_name _;

    root /var/www/html;
    index index.html;

    location /api/ {
        proxy_pass http://<BACKEND_PRIVATE_IP>:3000/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
```

Replace `<BACKEND_PRIVATE_IP>` with your backend EC2's private IP address.

### Restart Nginx

Restart Nginx to apply the configuration:

```bash
sudo systemctl restart nginx
```

### Test Connectivity

Optional: verify the frontend can reach the backend:

```bash
ping <BACKEND_PRIVATE_IP> -c 5
```

---

## Configure Backend (Node.js API)

This section covers setting up the Node.js backend API that connects to the ct-db database over Tailscale.

### Install Node.js and Dependencies

On the backend EC2 instance, install Node.js and create the application directory

```bash
sudo yum update -y && sudo yum upgrade -y
sudo yum install -y nodejs npm
mkdir ~/simple-app && cd ~/simple-app
npm init -y
npm install express body-parser cors mariadb
sudo yum install mariadb105 -y
```

### Create the Backend API (server.js)

Create `server.js` in the `~/simple-app` directory:

```javascript
const express = require('express');
const bodyParser = require('body-parser');
const mariadb = require('mariadb');
const cors = require('cors');
const path = require('path');

const app = express();

// Middleware
app.use(cors());
app.use(bodyParser.json());
app.use(express.static(path.join(__dirname, 'public'))); // Serve static files

// Create MariaDB pool with increased timeouts for VPN connection
const pool = mariadb.createPool({
  host: '192.168.8.101',
  user: 'appuser',
  password: 'StrongPassword123',
  database: 'appdb',
  port: 3306,
  connectionLimit: 10,
  connectTimeout: 30000,        // 30 seconds (default is 1000ms)
  acquireTimeout: 30000,         // 30 seconds for pool acquisition
  timeout: 30000,                // 30 seconds for queries
  socketTimeout: 30000,          // 30 seconds for socket operations
  initializationTimeout: 30000   // 30 seconds for pool initialization
});

// Test database connection on startup
(async () => {
  try {
    const conn = await pool.getConnection();
    console.log('✓ Database connected successfully');
    conn.release();
  } catch (err) {
    console.error('✗ Database connection failed:', err);
  }
})();

// API Routes
app.post('/api/register', async (req, res) => {
  const { username, password } = req.body;
  
  // Validation
  if (!username || !password) {
    return res.status(400).json({ 
      success: false, 
      message: 'Username and password required' 
    });
  }

  if (username.length < 3) {
    return res.status(400).json({ 
      success: false, 
      message: 'Username must be at least 3 characters' 
    });
  }

  if (password.length < 6) {
    return res.status(400).json({ 
      success: false, 
      message: 'Password must be at least 6 characters' 
    });
  }

  let conn;
  try {
    conn = await pool.getConnection();
    
    const result = await conn.query(
      'INSERT INTO users (username, password) VALUES (?, ?)',
      [username, password]
    );
    
    console.log(`✓ User registered: ${username} (ID: ${result.insertId})`);
    
    res.json({
      success: true,
      message: 'User registered successfully!',
      user: {
        username: username,
        id: result.insertId.toString()
      }
    });
    
  } catch (err) {
    console.error('Database error:', err);
    
    // Handle duplicate username
    if (err.code === 'ER_DUP_ENTRY') {
      return res.status(409).json({
        success: false,
        message: 'Username already exists'
      });
    }
    
    // Generic server error
    res.status(500).json({ 
      success: false, 
      message: 'Server error occurred' 
    });
    
  } finally {
    if (conn) conn.release();
  }
});

// Get all users (for testing/admin purposes)
app.get('/api/users', async (req, res) => {
  let conn;
  try {
    conn = await pool.getConnection();
    const rows = await conn.query('SELECT id, username, created_at FROM users ORDER BY created_at DESC');
    
    res.json({
      success: true,
      users: rows
    });
    
  } catch (err) {
    console.error('Database error:', err);
    res.status(500).json({ 
      success: false, 
      message: 'Server error' 
    });
  } finally {
    if (conn) conn.release();
  }
});

// Health check endpoint
app.get('/api/health', async (req, res) => {
  let dbStatus = 'disconnected';
  
  try {
    const conn = await pool.getConnection();
    dbStatus = 'connected';
    conn.release();
  } catch (err) {
    console.error('Health check failed:', err);
  }
  
  res.json({
    status: 'running',
    database: dbStatus,
    timestamp: new Date().toISOString()
  });
});

// 404 handler
app.use((req, res) => {
  res.status(404).json({ 
    success: false, 
    message: 'Endpoint not found' 
  });
});

// Error handler
app.use((err, req, res, next) => {
  console.error('Server error:', err);
  res.status(500).json({ 
    success: false, 
    message: 'Internal server error' 
  });
});

// Start server
const PORT = process.env.PORT || 3000;
app.listen(PORT, '0.0.0.0', () => {
  console.log('=================================');
  console.log(`🚀 Server running on port ${PORT}`);
  console.log(`📡 Access at: http://localhost:${PORT}`);
  console.log('=================================');
});
```

### Run the Backend Server

Start the Node.js server:

```bash
node server.js
```

You should see output like:

```plaintext
=================================
🚀 Server running on port 3000
📡 Access at: http://localhost:3000
=================================
```

---

## Create Two LXC Containers in Proxmox

This section covers creating two LXC containers on your Proxmox host: one for the database (ct-db) and one for monitoring (ct-monitor).

### Download LXC Template

In the Proxmox web UI:

1. Navigate to **Datacenter → proxmox → local (storage) → Templates**
    
2. Click **Templates** button
    
3. Download a Debian or Ubuntu template (e.g., `ubuntu-22.04-standard`)
    

### Create ct-db Container (Database)

Click **Create CT** and configure:

| Field | Value |
| --- | --- |
| Node | proxmox |
| CT ID | 101 |
| Hostname | ct-db |
| Password | (choose a secure password) |
| Template | Debian/Ubuntu template downloaded above |
| Disk Size | 10 GB |
| CPU | 2 cores |
| Memory | 2048 MB (2 GB) |
| Swap | 512 MB |
| Network - Bridge | vmbr0 |
| IPv4 | Static |
| IPv4/CIDR | 192.168.8.101/24 |
| Gateway | 192.168.8.1 *(Router Gateway Address)* |
| DNS Server | 8.8.8.8 |

After creation, enable "Start at boot" in ct-db → Options.

### Create ct-monitor Container (Monitoring)

Click **Create CT** and configure:

| Field | Value |
| --- | --- |
| Node | proxmox |
| CT ID | 102 |
| Hostname | ct-monitor |
| Password | (choose a secure password) |
| Template | Debian/Ubuntu template downloaded above |
| Disk Size | 10 GB |
| CPU | 2 cores |
| Memory | 2048 MB (2 GB) |
| Swap | 512 MB |
| Network - Bridge | vmbr0 |
| IPv4 | Static |
| IPv4/CIDR | 192.168.8.102/24 |
| Gateway | 192.168.8.1 *(Router Gateway Address)* |
| DNS Server | 8.8.8.8 |

After creation, enable "Start at boot" in ct-monitor → Options.

---

## Configure Database on ct-db

This section covers installing MariaDB, creating the database and user, and setting up the users table.

### Install and Start MariaDB

Access the ct-db container console from Proxmox UI or use:

```bash
pct enter 101
```

Update and install MariaDB:

```bash
sudo apt update && sudo apt upgrade -y
sudo apt install -y mariadb-server
sudo systemctl enable mariadb
sudo systemctl start mariadb
```

### Create Database and User

Log into MariaDB:

```bash
mysql -u root -p
```

Run the following SQL commands:

```sql
-- Create database
CREATE DATABASE appdb;

-- Show current users and their hosts
SELECT User, Host FROM mysql.user;

-- Create user with remote access
GRANT ALL PRIVILEGES ON appdb.* TO 'appuser'@'%' IDENTIFIED BY 'StrongPassword123';
FLUSH PRIVILEGES;

-- Verify the grant
SELECT User, Host FROM mysql.user WHERE User='appuser';

-- Switch to database
USE appdb;

-- Create users table
CREATE TABLE IF NOT EXISTS users (
  id INT AUTO_INCREMENT PRIMARY KEY,
  username VARCHAR(255) UNIQUE NOT NULL,
  password VARCHAR(255) NOT NULL,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Verify table structure
DESCRIBE users;

EXIT;
```

### Configure Remote Access

Edit the MariaDB configuration to allow remote connections:

```bash
sudo nano /etc/mysql/mariadb.conf.d/50-server.cnf
```

Find the `bind-address` line and change it to:

```plaintext
bind-address = 0.0.0.0
```

Restart MariaDB:

```bash
sudo systemctl restart mariadb
```

## Configure Monitoring Setup on ct-monitor (Prometheus + Grafana + Node Exporter)

This comprehensive section covers installing and configuring Prometheus, Grafana, and Node Exporter across all machines.

### 1\. Install on ct-monitor (192.168.8.102)

Access the ct-monitor container:

```bash
pct enter 102
```

#### Install Prometheus

```bash
# Access container
pct enter 102

# Update system
apt update && apt upgrade -y

# === PROMETHEUS ===
useradd --no-create-home --shell /bin/false prometheus
cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.47.0/prometheus-2.47.0.linux-amd64.tar.gz
tar -xvf prometheus-2.47.0.linux-amd64.tar.gz
cd prometheus-2.47.0.linux-amd64
mv prometheus promtool /usr/local/bin/
mkdir -p /etc/prometheus /var/lib/prometheus
mv consoles console_libraries /etc/prometheus/
chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

# Create Prometheus config (only 2 targets)
cat > /etc/prometheus/prometheus.yml <<EOF
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'ct-monitor'
    static_configs:
      - targets: ['localhost:9100']

  - job_name: 'backend-ec2'
    static_configs:
      - targets: ['BACKEND_TAILSCALE_IP:9100']
EOF

chown prometheus:prometheus /etc/prometheus/prometheus.yml

# Create Prometheus service
cat > /etc/systemd/system/prometheus.service <<EOF
[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl start prometheus
systemctl enable prometheus
```

#### Install Node Exporter

```bash
cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar -xvf node_exporter-1.6.1.linux-amd64.tar.gz
mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
useradd --no-create-home --shell /bin/false node_exporter

cat > /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl start node_exporter
systemctl enable node_exporter
```

Install **Grafana**

```bash
# === GRAFANA ===
apt install -y software-properties-common apt-transport-https wget
wget -q -O - https://packages.grafana.com/gpg.key | apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | tee /etc/apt/sources.list.d/grafana.list
apt update
apt install -y grafana

systemctl start grafana-server
systemctl enable grafana-server

# Verify all services
systemctl status prometheus
systemctl status node_exporter
systemctl status grafana-server
```

### 2\. Install on Backend EC2

```bash
# SSH to Backend EC2
cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar -xvf node_exporter-1.6.1.linux-amd64.tar.gz
sudo mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
sudo useradd --no-create-home --shell /bin/false node_exporter

sudo tee /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
sudo systemctl status node_exporter

# Get Tailscale IP for Prometheus config
tailscale ip -4
```

### 3\. Update Prometheus Config with Tailscale IPs

```bash
# On ct-monitor, get Tailscale IPs from EC2 instances
# Then edit Prometheus config
nano /etc/prometheus/prometheus.yml

# Replace BACKEND_TAILSCALE_IP and FRONTEND_TAILSCALE_IP with actual IPs
# Example:
#   - targets: ['100.117.212.126:9100']  # Backend
#   - targets: ['100.124.182.20:9100']   # Frontend

# Restart Prometheus
systemctl restart prometheus
```

### 4\. Access Grafana

```bash
# Open browser
http://192.168.8.102:3000

# Login: admin / admin
# Change password when prompted
```

### 5\. Configure Grafana Data Source

1. ### Login to Grafana
    
2. Go to **Configuration** → **Data Sources**
    
3. Click **Add data source**
    
4. Select **Prometheus**
    
5. URL: [`http://localhost:9090`](http://localhost:9090)
    
6. Click **Save & Test**
    

### 6\. Import Dashboard

1. Go to **Dashboards** → **Import**
    
2. Enter ID: `1860` (Node Exporter Full)
    
3. Click **Load**
    
4. Select Prometheus data source
    
5. Click **Import**
    

### Verification Commands

```bash
# Check all services on ct-monitor
systemctl status prometheus
systemctl status node_exporter
systemctl status grafana-server

# Check ports
ss -tulpn | grep -E '9090|9100|3000'

# Test from Backend EC2
curl http://192.168.8.102:9090
curl http://192.168.8.102:3000
```

---

## Proxmox host ↔ CT connectivity checks

#### Step 1 — Check container network interface (run on Proxmox host)

```bash
pct exec 101 -- ip addr
```

This shows the container’s eth0 IP and whether the interface is UP. If eth0 is missing or has no IP, it cannot respond.

#### Step 2 — Check container routing

```bash
pct exec 101 -- ip route
```

You should see something like:

```text
default via 192.168.8.1 dev eth0
192.168.8.0/24 dev eth0 proto kernel scope link src 192.168.8.101
```

#### Step 3 — Check bridge connectivity on host

```bash
brctl show vmbr0
ip addr show vmbr0
```

Confirm vmbr0 is UP and the container veth (veth101i0) is attached.

#### Step 4 — Restart container network stack

Sometimes the network inside a container doesn’t come up automatically after stop/start:

```bash
pct exec 101 -- ip link set eth0 up
pct exec 101 -- ip addr add 192.168.8.101/24 dev eth0
pct exec 101 -- ip route add default via 192.168.8.1
```

Adjust IP/gateway as per your container config.

Now test:

```bash
# From Proxmox host
ping 192.168.8.101 -c 3

# Inside container
ping 8.8.8.8 -c 3
```

---

## Backend EC2 ↔ ct-db over Tailscale

On Proxmox host (advertise the LXC subnet):

```bash
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --advertise-routes=192.168.8.0/24
```

On EC2 (accept advertised routes):

```bash
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --accept-routes
```

On backend EC2, validate routing and DB port reachability:

```bash
tailscale status
ip route get 192.168.8.101   # expect dev tailscale0
nc -zv 192.168.8.101 3306    # DB port test (MariaDB/MySQL)
```

Troubleshooting (see Article 1): ensure IP forwarding + NAT on Proxmox, and `--accept-routes` on EC2.

---

## Monitoring on ct-monitor (Prometheus + Grafana)

On ct-monitor:

1. Install Prometheus, run as a dedicated user, and expose on 0.0.0.0:9090
    
2. Configure scrape targets ([localhost](http://localhost) and any exporters)
    
3. Install Grafana via apt repo; enable grafana-server
    
4. Access Grafana from your LAN and add dashboards (system metrics, DB metrics, blackbox checks)
    

Your notes include working unit files and a minimal prometheus.yml; reuse them here.

---

## End-to-end test and verification

1. From frontend: open the site via public IP; navigate to the registration page
    
2. Submit a username/password; the request hits frontend `/api/register` and proxies to backend private IP:3000
    
3. Backend connects to ct-db at 192.168.8.101:3306 (over Tailscale) and inserts into users
    
4. Confirm in DB: SELECT \* FROM users; on ct-db
    
5. Optional: Create a Grafana dashboard to visualize backend host metrics
    

If requests time out or DB connection fails, check Article 1’s troubleshooting section (routing, NAT, bind-address, user grants).
