Hybrid Cloud Full-Stack App Deployment Guide

This guide covers deploying a simple full‑stack app with an AWS frontend and backend that talks to an on‑prem database running in a Proxmox LXC container, plus a basic monitoring stack. It assumes you’ve completed the network setup in Article 1 and can reach ct-db from EC2 over Tailscale.

Overview and architecture
Create AWS VPC (public + private subnets)
Provision EC2 instances (frontend public, backend private)
Configure frontend → backend communication (Nginx reverse proxy)
Create two LXC in Proxmox (ct-db, ct-monitor)
Proxmox host ↔ CT connectivity checks
Backend EC2 ↔ ct-db over Tailscale
Backend (Node.js + MariaDB client) setup
Database on ct-db (MariaDB recommended)
Monitoring on ct-monitor (Prometheus + Grafana)
End-to-end test and verification
Security hardening

Overview and architecture

High-level flow: A public frontend EC2 serves the website and proxies API requests to a private backend EC2. The backend talks to an on‑prem database (ct-db) over a Tailscale mesh to your Proxmox host. Monitoring runs in another LXC (ct-monitor).

Prereqs from Article 1:

Proxmox with two LXC containers
- ct-db: 192.168.8.101 (MariaDB/PostgreSQL)
- ct-monitor: 192.168.8.102 (Prometheus + Grafana)
Tailscale configured
- Proxmox advertises 192.168.8.0/24
- EC2 instances accept routes
Verified EC2 → ct-db connectivity (ping + TCP/3306)

Create AWS VPC (public + private subnets)

Minimal setup:

VPC with one public and one private subnet
- Public subnet: Internet Gateway + route for 0.0.0.0/0
- Private subnet: NAT Gateway + route for 0.0.0.0/0 via NAT
Route tables
- Associate the public subnet with the IGW route table
- Associate the private subnet with the NAT route table

Keep the backend private; you’ll reach it from the frontend using its private IP and an Nginx reverse proxy.

Provision EC2 instances (frontend public, backend private)

Create two EC2 instances in your VPC: one in the public subnet for the frontend and one in the private subnet for the backend.

Frontend EC2 (Public Subnet)

Configuration	Value
AMI	Amazon Linux 2023 or Ubuntu 22.04 LTS
Instance Type	t2.micro or t2.nano
Subnet	Public subnet
Auto-assign Public IP	Enable
Security Group	SG-frontend

SG-frontend Rules:

Port 22 (SSH) from your IP
Port 80 (HTTP) from 0.0.0.0/0
Port 443 (HTTPS) from 0.0.0.0/0

Backend EC2 (Private Subnet)

Configuration	Value
AMI	Amazon Linux 2023 or Ubuntu 22.04 LTS
Instance Type	t2.micro or t2.nano
Subnet	Private subnet
Auto-assign Public IP	Disable
Security Group	SG-backend

SG-backend Rules:

Port 22 (SSH) from SG-frontend
Port 3000 (TCP) from SG-frontend

SSH Access and Bastion Setup

Connect to Frontend EC2

Log into the frontend VM using SSH with your PEM key:

ssh -i hybrid-cloud.pem ec2-user@<FRONTEND_PUBLIC_IP>

Connect to Backend EC2 (via bastion)

The backend instance has no public IP, so use the frontend as a bastion host.

Step 1: Copy the PEM key to the frontend instance:

scp -i hybrid-cloud.pem hybrid-cloud.pem ec2-user@<FRONTEND_PUBLIC_IP>:/home/ec2-user/

Step 2: SSH from the frontend to the backend using its private IP:

ssh -i ~/hybrid-cloud.pem ec2-user@<BACKEND_PRIVATE_IP>

Configure Frontend (Nginx + Static Site + Reverse Proxy)

This section covers installing Nginx, creating the static HTML pages, and configuring the reverse proxy to route API requests to the backend.

Install and Enable Nginx

Update packages and install Nginx:

sudo yum update -y && sudo yum upgrade -y
sudo yum install -y nginx
sudo systemctl enable --now nginx

Create Web Root and HTML Files

Create the web root directory:

sudo mkdir -p /var/www/html
cd /var/www/html

Create index.html:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Hybrid Cloud Platform</title>
  <style>
    * {
      margin: 0;
      padding: 0;
      box-sizing: border-box;
    }

    body {
      font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
      background: linear-gradient(135deg, #31694E 0%, #658C58 100%);
      min-height: 100vh;
      display: flex;
      justify-content: center;
      align-items: center;
      padding: 20px;
    }

    .container {
      background: #F0E491;
      border-radius: 20px;
      box-shadow: 0 20px 60px rgba(0, 0, 0, 0.3);
      padding: 60px 40px;
      max-width: 600px;
      width: 100%;
      text-align: center;
    }

    h1 {
      color: #31694E;
      font-size: 2.5em;
      margin-bottom: 15px;
      font-weight: 700;
    }

    .subtitle {
      color: #658C58;
      font-size: 1.2em;
      margin-bottom: 40px;
      font-weight: 500;
    }

    .project-scope {
      background: white;
      border-left: 5px solid #BBC863;
      padding: 20px;
      margin: 30px 0;
      border-radius: 8px;
      text-align: left;
    }

    .project-scope h2 {
      color: #31694E;
      font-size: 1.3em;
      margin-bottom: 10px;
    }

    .project-scope p {
      color: #333;
      line-height: 1.6;
      font-size: 1.05em;
    }

    .tech-badges {
      display: flex;
      justify-content: center;
      gap: 15px;
      margin: 20px 0;
      flex-wrap: wrap;
    }

    .badge {
      background: #658C58;
      color: #F0E491;
      padding: 8px 20px;
      border-radius: 20px;
      font-size: 0.9em;
      font-weight: 600;
    }

    .cta-button {
      display: inline-block;
      background: #31694E;
      color: #F0E491;
      text-decoration: none;
      padding: 18px 50px;
      border-radius: 50px;
      font-size: 1.3em;
      font-weight: 700;
      margin-top: 30px;
      transition: all 0.3s ease;
      box-shadow: 0 8px 20px rgba(49, 105, 78, 0.4);
      position: relative;
      overflow: hidden;
    }

    .cta-button::before {
      content: '';
      position: absolute;
      top: 0;
      left: -100%;
      width: 100%;
      height: 100%;
      background: linear-gradient(90deg, transparent, rgba(255, 255, 255, 0.2), transparent);
      transition: left 0.5s;
    }

    .cta-button:hover::before {
      left: 100%;
    }

    .cta-button:hover {
      background: #254d3a;
      transform: translateY(-3px);
      box-shadow: 0 12px 30px rgba(49, 105, 78, 0.5);
    }

    .highlight-box {
      background: #BBC863;
      color: #31694E;
      padding: 5px 15px;
      border-radius: 5px;
      display: inline-block;
      font-weight: 700;
      margin-top: 10px;
      font-size: 0.95em;
    }

    @media (max-width: 600px) {
      .container {
        padding: 40px 25px;
      }

      h1 {
        font-size: 2em;
      }

      .subtitle {
        font-size: 1em;
      }
    }
  </style>
</head>
<body>
  <div class="container">
    <h1>🚀 Hybrid Cloud Platform</h1>
    <p class="subtitle">Seamless Integration Across Environments</p>

    <div class="project-scope">
      <h2>Project Scope</h2>
      <p><strong>Hybrid Cloud Infrastructure:</strong> AWS Cloud + Proxmox Homelab</p>
      <div class="tech-badges">
        <span class="badge">AWS</span>
        <span class="badge">Proxmox</span>
        <span class="badge">Homelab</span>
      </div>
    </div>

    <a href="register.html" class="cta-button">Register as a User</a>
  </div>
</body>
</html>

Create register.html:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Register - Hybrid Cloud Platform</title>
  <style>
    * {
      margin: 0;
      padding: 0;
      box-sizing: border-box;
    }

    body {
      font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
      background: linear-gradient(135deg, #31694E 0%, #658C58 100%);
      min-height: 100vh;
      display: flex;
      justify-content: center;
      align-items: center;
      padding: 20px;
    }

    .container {
      background: #F0E491;
      border-radius: 20px;
      box-shadow: 0 20px 60px rgba(0, 0, 0, 0.3);
      padding: 50px 40px;
      max-width: 500px;
      width: 100%;
    }

    .header {
      text-align: center;
      margin-bottom: 40px;
    }

    h2 {
      color: #31694E;
      font-size: 2.2em;
      margin-bottom: 10px;
      font-weight: 700;
    }

    .subtitle {
      color: #658C58;
      font-size: 1em;
      margin-bottom: 10px;
    }

    .back-link {
      display: inline-block;
      color: #31694E;
      text-decoration: none;
      font-size: 0.95em;
      margin-bottom: 20px;
      transition: color 0.3s;
    }

    .back-link:hover {
      color: #658C58;
    }

    .form-group {
      margin-bottom: 25px;
    }

    label {
      display: block;
      color: #31694E;
      font-weight: 600;
      margin-bottom: 8px;
      font-size: 0.95em;
    }

    input[type="text"],
    input[type="password"] {
      width: 100%;
      padding: 15px;
      border: 2px solid #BBC863;
      border-radius: 10px;
      font-size: 1em;
      background: white;
      color: #31694E;
      transition: all 0.3s;
      outline: none;
    }

    input[type="text"]:focus,
    input[type="password"]:focus {
      border-color: #658C58;
      box-shadow: 0 0 0 3px rgba(101, 140, 88, 0.1);
    }

    input::placeholder {
      color: #BBC863;
    }

    .btn-submit {
      width: 100%;
      padding: 16px;
      background: #31694E;
      color: #F0E491;
      border: none;
      border-radius: 10px;
      font-size: 1.1em;
      font-weight: 700;
      cursor: pointer;
      transition: all 0.3s;
      box-shadow: 0 6px 20px rgba(49, 105, 78, 0.3);
      margin-top: 10px;
    }

    .btn-submit:hover {
      background: #254d3a;
      transform: translateY(-2px);
      box-shadow: 0 8px 25px rgba(49, 105, 78, 0.4);
    }

    .btn-submit:active {
      transform: translateY(0);
    }

    .btn-submit:disabled {
      background: #BBC863;
      cursor: not-allowed;
      transform: none;
    }

    .info-box {
      background: white;
      border-left: 4px solid #BBC863;
      padding: 15px;
      border-radius: 8px;
      margin-top: 25px;
      font-size: 0.9em;
      color: #31694E;
    }

    .alert {
      padding: 15px;
      border-radius: 8px;
      margin-top: 20px;
      font-weight: 500;
      display: none;
    }

    .alert.success {
      background: #BBC863;
      color: #31694E;
      display: block;
    }

    .alert.error {
      background: #ff6b6b;
      color: white;
      display: block;
    }

    @media (max-width: 600px) {
      .container {
        padding: 35px 25px;
      }

      h2 {
        font-size: 1.8em;
      }
    }
  </style>
</head>
<body>
  <div class="container">
    <a href="index.html" class="back-link">← Back to Home</a>

    <div class="header">
      <h2>Create Account</h2>
      <p class="subtitle">Join the Hybrid Cloud Platform</p>
    </div>

    <form id="regForm">
      <div class="form-group">
        <label for="username">Username</label>
        <input type="text" id="username" placeholder="Enter your username" required>
      </div>

      <div class="form-group">
        <label for="password">Password</label>
        <input type="password" id="password" placeholder="Enter your password" required>
      </div>

      <button type="submit" class="btn-submit">Register Account</button>

      <div id="alertBox" class="alert"></div>
    </form>

    <div class="info-box">
      <strong>🔒 Secure Registration</strong><br>
      Your credentials will be securely stored in our hybrid cloud infrastructure.
    </div>
  </div>

  <script>
    document.getElementById('regForm').addEventListener('submit', async (e) => {
      e.preventDefault();

      const alertBox = document.getElementById('alertBox');
      const submitBtn = e.target.querySelector('.btn-submit');
      const username = document.getElementById('username').value;
      const password = document.getElementById('password').value;

      // Reset alert
      alertBox.className = 'alert';
      alertBox.textContent = '';

      // Disable button during request
      submitBtn.disabled = true;
      submitBtn.textContent = 'Registering...';

      try {
        const res = await fetch('/api/register', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify({ username, password })
        });

        // Check if response is ok
        if (!res.ok) {
          throw new Error(`Server returned ${res.status}: ${res.statusText}`);
        }

        // Check if response is JSON
        const contentType = res.headers.get('content-type');
        if (!contentType || !contentType.includes('application/json')) {
          const text = await res.text();
          throw new Error(`Expected JSON but got: ${text.substring(0, 100)}`);
        }

        const data = await res.json();

        if (data.success) {
          alertBox.className = 'alert success';
          alertBox.textContent = `✓ User ${data.user.username} registered successfully!`;
          document.getElementById('regForm').reset();
        } else {
          alertBox.className = 'alert error';
          alertBox.textContent = `✗ Error: ${data.message}`;
        }
      } catch (error) {
        alertBox.className = 'alert error';
        alertBox.textContent = `✗ Error: ${error.message}`;
        console.error('Registration error:', error);
      } finally {
        // Re-enable button
        submitBtn.disabled = false;
        submitBtn.textContent = 'Register Account';
      }
    });
  </script>
</body>
</html>

Configure Nginx Reverse Proxy

Edit the Nginx configuration (Amazon Linux path: /etc/nginx/nginx.conf or create a file under /etc/nginx/conf.d/):

server {
    listen 80;
    server_name _;

    root /var/www/html;
    index index.html;

    location /api/ {
        proxy_pass http://<BACKEND_PRIVATE_IP>:3000/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Replace <BACKEND_PRIVATE_IP> with your backend EC2's private IP address.

Restart Nginx

Restart Nginx to apply the configuration:

sudo systemctl restart nginx

Test Connectivity

Optional: verify the frontend can reach the backend:

ping <BACKEND_PRIVATE_IP> -c 5

Configure Backend (Node.js API)

This section covers setting up the Node.js backend API that connects to the ct-db database over Tailscale.

Install Node.js and Dependencies

On the backend EC2 instance, install Node.js and create the application directory

sudo yum update -y && sudo yum upgrade -y
sudo yum install -y nodejs npm
mkdir ~/simple-app && cd ~/simple-app
npm init -y
npm install express body-parser cors mariadb
sudo yum install mariadb105 -y

Create the Backend API (server.js)

Create server.js in the ~/simple-app directory:

const express = require('express');
const bodyParser = require('body-parser');
const mariadb = require('mariadb');
const cors = require('cors');
const path = require('path');

const app = express();

// Middleware
app.use(cors());
app.use(bodyParser.json());
app.use(express.static(path.join(__dirname, 'public'))); // Serve static files

// Create MariaDB pool with increased timeouts for VPN connection
const pool = mariadb.createPool({
  host: '192.168.8.101',
  user: 'appuser',
  password: 'StrongPassword123',
  database: 'appdb',
  port: 3306,
  connectionLimit: 10,
  connectTimeout: 30000,        // 30 seconds (default is 1000ms)
  acquireTimeout: 30000,         // 30 seconds for pool acquisition
  timeout: 30000,                // 30 seconds for queries
  socketTimeout: 30000,          // 30 seconds for socket operations
  initializationTimeout: 30000   // 30 seconds for pool initialization
});

// Test database connection on startup
(async () => {
  try {
    const conn = await pool.getConnection();
    console.log('✓ Database connected successfully');
    conn.release();
  } catch (err) {
    console.error('✗ Database connection failed:', err);
  }
})();

// API Routes
app.post('/api/register', async (req, res) => {
  const { username, password } = req.body;

  // Validation
  if (!username || !password) {
    return res.status(400).json({ 
      success: false, 
      message: 'Username and password required' 
    });
  }

  if (username.length < 3) {
    return res.status(400).json({ 
      success: false, 
      message: 'Username must be at least 3 characters' 
    });
  }

  if (password.length < 6) {
    return res.status(400).json({ 
      success: false, 
      message: 'Password must be at least 6 characters' 
    });
  }

  let conn;
  try {
    conn = await pool.getConnection();

    const result = await conn.query(
      'INSERT INTO users (username, password) VALUES (?, ?)',
      [username, password]
    );

    console.log(`✓ User registered: ${username} (ID: ${result.insertId})`);

    res.json({
      success: true,
      message: 'User registered successfully!',
      user: {
        username: username,
        id: result.insertId.toString()
      }
    });

  } catch (err) {
    console.error('Database error:', err);

    // Handle duplicate username
    if (err.code === 'ER_DUP_ENTRY') {
      return res.status(409).json({
        success: false,
        message: 'Username already exists'
      });
    }

    // Generic server error
    res.status(500).json({ 
      success: false, 
      message: 'Server error occurred' 
    });

  } finally {
    if (conn) conn.release();
  }
});

// Get all users (for testing/admin purposes)
app.get('/api/users', async (req, res) => {
  let conn;
  try {
    conn = await pool.getConnection();
    const rows = await conn.query('SELECT id, username, created_at FROM users ORDER BY created_at DESC');

    res.json({
      success: true,
      users: rows
    });

  } catch (err) {
    console.error('Database error:', err);
    res.status(500).json({ 
      success: false, 
      message: 'Server error' 
    });
  } finally {
    if (conn) conn.release();
  }
});

// Health check endpoint
app.get('/api/health', async (req, res) => {
  let dbStatus = 'disconnected';

  try {
    const conn = await pool.getConnection();
    dbStatus = 'connected';
    conn.release();
  } catch (err) {
    console.error('Health check failed:', err);
  }

  res.json({
    status: 'running',
    database: dbStatus,
    timestamp: new Date().toISOString()
  });
});

// 404 handler
app.use((req, res) => {
  res.status(404).json({ 
    success: false, 
    message: 'Endpoint not found' 
  });
});

// Error handler
app.use((err, req, res, next) => {
  console.error('Server error:', err);
  res.status(500).json({ 
    success: false, 
    message: 'Internal server error' 
  });
});

// Start server
const PORT = process.env.PORT || 3000;
app.listen(PORT, '0.0.0.0', () => {
  console.log('=================================');
  console.log(`🚀 Server running on port ${PORT}`);
  console.log(`📡 Access at: http://localhost:${PORT}`);
  console.log('=================================');
});

Run the Backend Server

Start the Node.js server:

node server.js

You should see output like:

=================================
🚀 Server running on port 3000
📡 Access at: http://localhost:3000
=================================

Create Two LXC Containers in Proxmox

This section covers creating two LXC containers on your Proxmox host: one for the database (ct-db) and one for monitoring (ct-monitor).

Download LXC Template

In the Proxmox web UI:

Navigate to Datacenter → proxmox → local (storage) → Templates
Click Templates button
Download a Debian or Ubuntu template (e.g., ubuntu-22.04-standard)

Create ct-db Container (Database)

Click Create CT and configure:

Field	Value
Node	proxmox
CT ID	101
Hostname	ct-db
Password	(choose a secure password)
Template	Debian/Ubuntu template downloaded above
Disk Size	10 GB
CPU	2 cores
Memory	2048 MB (2 GB)
Swap	512 MB
Network - Bridge	vmbr0
IPv4	Static
IPv4/CIDR	192.168.8.101/24
Gateway	192.168.8.1 (Router Gateway Address)
DNS Server	8.8.8.8

After creation, enable "Start at boot" in ct-db → Options.

Create ct-monitor Container (Monitoring)

Click Create CT and configure:

Field	Value
Node	proxmox
CT ID	102
Hostname	ct-monitor
Password	(choose a secure password)
Template	Debian/Ubuntu template downloaded above
Disk Size	10 GB
CPU	2 cores
Memory	2048 MB (2 GB)
Swap	512 MB
Network - Bridge	vmbr0
IPv4	Static
IPv4/CIDR	192.168.8.102/24
Gateway	192.168.8.1 (Router Gateway Address)
DNS Server	8.8.8.8

After creation, enable "Start at boot" in ct-monitor → Options.

Configure Database on ct-db

This section covers installing MariaDB, creating the database and user, and setting up the users table.

Install and Start MariaDB

Access the ct-db container console from Proxmox UI or use:

pct enter 101

Update and install MariaDB:

sudo apt update && sudo apt upgrade -y
sudo apt install -y mariadb-server
sudo systemctl enable mariadb
sudo systemctl start mariadb

Create Database and User

Log into MariaDB:

mysql -u root -p

Run the following SQL commands:

-- Create database
CREATE DATABASE appdb;

-- Show current users and their hosts
SELECT User, Host FROM mysql.user;

-- Create user with remote access
GRANT ALL PRIVILEGES ON appdb.* TO 'appuser'@'%' IDENTIFIED BY 'StrongPassword123';
FLUSH PRIVILEGES;

-- Verify the grant
SELECT User, Host FROM mysql.user WHERE User='appuser';

-- Switch to database
USE appdb;

-- Create users table
CREATE TABLE IF NOT EXISTS users (
  id INT AUTO_INCREMENT PRIMARY KEY,
  username VARCHAR(255) UNIQUE NOT NULL,
  password VARCHAR(255) NOT NULL,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Verify table structure
DESCRIBE users;

EXIT;

Configure Remote Access

Edit the MariaDB configuration to allow remote connections:

sudo nano /etc/mysql/mariadb.conf.d/50-server.cnf

Find the bind-address line and change it to:

bind-address = 0.0.0.0

Restart MariaDB:

sudo systemctl restart mariadb

Configure Monitoring Setup on ct-monitor (Prometheus + Grafana + Node Exporter)

This comprehensive section covers installing and configuring Prometheus, Grafana, and Node Exporter across all machines.

1. Install on ct-monitor (192.168.8.102)

Access the ct-monitor container:

pct enter 102

Install Prometheus

# Access container
pct enter 102

# Update system
apt update && apt upgrade -y

# === PROMETHEUS ===
useradd --no-create-home --shell /bin/false prometheus
cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.47.0/prometheus-2.47.0.linux-amd64.tar.gz
tar -xvf prometheus-2.47.0.linux-amd64.tar.gz
cd prometheus-2.47.0.linux-amd64
mv prometheus promtool /usr/local/bin/
mkdir -p /etc/prometheus /var/lib/prometheus
mv consoles console_libraries /etc/prometheus/
chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

# Create Prometheus config (only 2 targets)
cat > /etc/prometheus/prometheus.yml <<EOF
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'ct-monitor'
    static_configs:
      - targets: ['localhost:9100']

  - job_name: 'backend-ec2'
    static_configs:
      - targets: ['BACKEND_TAILSCALE_IP:9100']
EOF

chown prometheus:prometheus /etc/prometheus/prometheus.yml

# Create Prometheus service
cat > /etc/systemd/system/prometheus.service <<EOF
[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl start prometheus
systemctl enable prometheus

Install Node Exporter

cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar -xvf node_exporter-1.6.1.linux-amd64.tar.gz
mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
useradd --no-create-home --shell /bin/false node_exporter

cat > /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl start node_exporter
systemctl enable node_exporter

Install Grafana

# === GRAFANA ===
apt install -y software-properties-common apt-transport-https wget
wget -q -O - https://packages.grafana.com/gpg.key | apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | tee /etc/apt/sources.list.d/grafana.list
apt update
apt install -y grafana

systemctl start grafana-server
systemctl enable grafana-server

# Verify all services
systemctl status prometheus
systemctl status node_exporter
systemctl status grafana-server

2. Install on Backend EC2

# SSH to Backend EC2
cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar -xvf node_exporter-1.6.1.linux-amd64.tar.gz
sudo mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
sudo useradd --no-create-home --shell /bin/false node_exporter

sudo tee /etc/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
sudo systemctl status node_exporter

# Get Tailscale IP for Prometheus config
tailscale ip -4

3. Update Prometheus Config with Tailscale IPs

# On ct-monitor, get Tailscale IPs from EC2 instances
# Then edit Prometheus config
nano /etc/prometheus/prometheus.yml

# Replace BACKEND_TAILSCALE_IP and FRONTEND_TAILSCALE_IP with actual IPs
# Example:
#   - targets: ['100.117.212.126:9100']  # Backend
#   - targets: ['100.124.182.20:9100']   # Frontend

# Restart Prometheus
systemctl restart prometheus

4. Access Grafana

# Open browser
http://192.168.8.102:3000

# Login: admin / admin
# Change password when prompted

5. Configure Grafana Data Source

Login to Grafana
Go to Configuration → Data Sources
Click Add data source
Select Prometheus
URL: http://localhost:9090
Click Save & Test

6. Import Dashboard

Go to Dashboards → Import
Enter ID: 1860 (Node Exporter Full)
Click Load
Select Prometheus data source
Click Import

Verification Commands

# Check all services on ct-monitor
systemctl status prometheus
systemctl status node_exporter
systemctl status grafana-server

# Check ports
ss -tulpn | grep -E '9090|9100|3000'

# Test from Backend EC2
curl http://192.168.8.102:9090
curl http://192.168.8.102:3000

Proxmox host ↔ CT connectivity checks

Step 1 — Check container network interface (run on Proxmox host)

pct exec 101 -- ip addr

This shows the container’s eth0 IP and whether the interface is UP. If eth0 is missing or has no IP, it cannot respond.

Step 2 — Check container routing

pct exec 101 -- ip route

You should see something like:

default via 192.168.8.1 dev eth0
192.168.8.0/24 dev eth0 proto kernel scope link src 192.168.8.101

Step 3 — Check bridge connectivity on host

brctl show vmbr0
ip addr show vmbr0

Confirm vmbr0 is UP and the container veth (veth101i0) is attached.

Step 4 — Restart container network stack

Sometimes the network inside a container doesn’t come up automatically after stop/start:

pct exec 101 -- ip link set eth0 up
pct exec 101 -- ip addr add 192.168.8.101/24 dev eth0
pct exec 101 -- ip route add default via 192.168.8.1

Adjust IP/gateway as per your container config.

Now test:

# From Proxmox host
ping 192.168.8.101 -c 3

# Inside container
ping 8.8.8.8 -c 3

Backend EC2 ↔ ct-db over Tailscale

On Proxmox host (advertise the LXC subnet):

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --advertise-routes=192.168.8.0/24

On EC2 (accept advertised routes):

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --accept-routes

On backend EC2, validate routing and DB port reachability:

tailscale status
ip route get 192.168.8.101   # expect dev tailscale0
nc -zv 192.168.8.101 3306    # DB port test (MariaDB/MySQL)

Troubleshooting (see Article 1): ensure IP forwarding + NAT on Proxmox, and --accept-routes on EC2.

Monitoring on ct-monitor (Prometheus + Grafana)

On ct-monitor:

Install Prometheus, run as a dedicated user, and expose on 0.0.0.0:9090
Configure scrape targets (localhost and any exporters)
Install Grafana via apt repo; enable grafana-server
Access Grafana from your LAN and add dashboards (system metrics, DB metrics, blackbox checks)

Your notes include working unit files and a minimal prometheus.yml; reuse them here.

End-to-end test and verification

From frontend: open the site via public IP; navigate to the registration page
Submit a username/password; the request hits frontend /api/register and proxies to backend private IP:3000
Backend connects to ct-db at 192.168.8.101:3306 (over Tailscale) and inserts into users
Confirm in DB: SELECT * FROM users; on ct-db
Optional: Create a Grafana dashboard to visualize backend host metrics

If requests time out or DB connection fails, check Article 1’s troubleshooting section (routing, NAT, bind-address, user grants).

Command Palette

Table of contents