HOA_Financial_Platform/docs/DEPLOYMENT.md

# HOA LedgerIQ — Deployment Guide

**Version:** 2026.3.2 (beta)
**Last updated:** 2026-03-02

---

## Table of Contents

1. [Prerequisites](#prerequisites)
2. [Deploy to a Fresh Docker Server](#deploy-to-a-fresh-docker-server)
3. [Production Deployment](#production-deployment)
4. [SSL with Certbot (Let's Encrypt)](#ssl-with-certbot-lets-encrypt)
5. [Backup the Local Test Database](#backup-the-local-test-database)
6. [Restore a Backup into the Staged Environment](#restore-a-backup-into-the-staged-environment)
7. [Running Migrations on the Staged Environment](#running-migrations-on-the-staged-environment)
8. [Verifying the Deployment](#verifying-the-deployment)
9. [Environment Variable Reference](#environment-variable-reference)

---

## Prerequisites

On the **target server**, ensure the following are installed:

| Tool            | Minimum Version |
|-----------------|-----------------|
| Docker Engine   | 24+             |
| Docker Compose  | v2+             |
| Git             | 2.x             |
| `psql` (client) | 15+ *(optional, for manual DB work)* |

The app runs four containers in production — backend (NestJS), frontend
(React/nginx), PostgreSQL 15, and Redis 7. A fifth nginx container is used
in dev mode only. Total memory footprint is roughly **1–2 GB** idle.

For SSL, the server must also have:
- A **public hostname** with a DNS A record pointing to the server's IP
  (e.g., `staging.yourdomain.com → 203.0.113.10`)
- **Ports 80 and 443** open in any firewall / security group

---

## Deploy to a Fresh Docker Server

### 1. Clone the repository

```bash
ssh your-staging-server

git clone <repo-url> /opt/hoa-ledgeriq
cd /opt/hoa-ledgeriq
```

### 2. Create the environment file

Copy the example and fill in real values:

```bash
cp .env.example .env
nano .env          # or vi, your choice
```

**Required changes from defaults:**

```dotenv
# --- CHANGE THESE ---
POSTGRES_PASSWORD=<strong-random-password>
JWT_SECRET=<random-64-char-string>

# Database URL must match the password above
DATABASE_URL=postgresql://hoafinance:<same-password>@postgres:5432/hoafinance

# AI features (get a key from build.nvidia.com)
AI_API_KEY=nvapi-xxxxxxxxxxxx

# --- Usually fine as-is ---
POSTGRES_USER=hoafinance
POSTGRES_DB=hoafinance
REDIS_URL=redis://redis:6379
NODE_ENV=development        # keep as development for staging
AI_API_URL=https://integrate.api.nvidia.com/v1
AI_MODEL=qwen/qwen3.5-397b-a17b
AI_DEBUG=false
```

> **Tip:** Generate secrets quickly:
> ```bash
> openssl rand -hex 32    # good for JWT_SECRET
> openssl rand -base64 24 # good for POSTGRES_PASSWORD
> ```

### 3. Build and start the stack

```bash
docker compose up -d --build
```

This will:
- Build the backend and frontend images
- Pull `postgres:15-alpine`, `redis:7-alpine`, and `nginx:alpine`
- Initialize the PostgreSQL database with the shared schema (`db/init/00-init.sql`)
- Start all services on the `hoanet` bridge network

### 4. Wait for healthy services

```bash
docker compose ps
```

All containers should show `Up` (postgres and redis should also show
`(healthy)`). If the backend is restarting, check logs:

```bash
docker compose logs backend --tail=50
```

### 5. (Optional) Seed with demo data

If deploying a fresh environment for testing and you want the Sunrise Valley
HOA demo tenant:

```bash
docker compose exec -T postgres psql -U hoafinance -d hoafinance < db/seed/seed.sql
```

This creates:
- Platform admin: `admin@hoaledgeriq.com` / `password123`
- Tenant admin: `admin@sunrisevalley.org` / `password123`
- Tenant viewer: `viewer@sunrisevalley.org` / `password123`

### 6. Access the application

| Service   | URL                            |
|-----------|--------------------------------|
| App (UI)  | `http://<server-ip>`           |
| API       | `http://<server-ip>/api`       |
| Postgres  | `<server-ip>:5432` (direct)    |

> At this point the app is running over **plain HTTP** in development mode.
> For any environment that will serve real traffic, continue to the Production
> Deployment section.

---

## Production Deployment

The base `docker-compose.yml` runs everything in **development mode** (Vite
dev server, NestJS in watch mode, no connection pooling). This is fine for
local development but will fail under even light production load.

`docker-compose.prod.yml` provides a production overlay that fixes this:

| Component | Dev mode | Production mode |
|-----------|----------|-----------------|
| Frontend  | Vite dev server (single-threaded, HMR) | Static build served by nginx |
| Backend   | `nest start --watch` (ts-node, file watcher) | Compiled JS, clustered across CPU cores |
| DB pooling | None (new connection per query) | Pool of 30 reusable connections |
| Postgres  | Default config (100 connections) | Tuned: 200 connections, optimized buffers |
| Nginx     | Docker nginx routes all traffic | Disabled — host nginx routes directly |
| Restart   | None | `unless-stopped` on all services |

### Deploy for production

```bash
cd /opt/hoa-ledgeriq

# Ensure .env has NODE_ENV=production and strong secrets
nano .env

# Build and start with the production overlay
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build
```

The production overlay **disables the Docker nginx container** — request routing
and SSL are handled by the host-level nginx. Backend and frontend are exposed
on `127.0.0.1` only (loopback), so they aren't publicly accessible without the
host nginx in front.

### Host nginx setup (required for production)

A ready-to-use host nginx config is included at `nginx/host-production.conf`.
It handles SSL termination, request routing, rate limiting, proxy buffering,
and extended timeouts for AI endpoints.

```bash
# Copy the reference config
sudo cp nginx/host-production.conf /etc/nginx/sites-available/app.yourdomain.com

# Edit the hostname (replace all instances of app.yourdomain.com)
sudo sed -i 's/app.yourdomain.com/YOUR_HOSTNAME/g' \
  /etc/nginx/sites-available/app.yourdomain.com

# Enable the site
sudo ln -s /etc/nginx/sites-available/app.yourdomain.com /etc/nginx/sites-enabled/

# Get an SSL certificate (certbot modifies the config automatically)
sudo certbot --nginx -d YOUR_HOSTNAME

# Test and reload
sudo nginx -t && sudo systemctl reload nginx
```

The host config routes traffic directly to the Docker services:
- `/api/*` → `http://127.0.0.1:3000` (NestJS backend)
- `/` → `http://127.0.0.1:3001` (React frontend served by nginx)

> See `nginx/host-production.conf` for the full config including rate limiting,
> proxy buffering, and extended AI endpoint timeouts.

> **Tip:** Create a shell alias to avoid typing the compose files every time:
> ```bash
> echo 'alias dc="docker compose -f docker-compose.yml -f docker-compose.prod.yml"' >> ~/.bashrc
> source ~/.bashrc
> dc up -d --build
> ```

### What the production overlay does

**Backend (`backend/Dockerfile`)**
- Multi-stage build: compiles TypeScript once, runs `node dist/main`
- No dev dependencies shipped (smaller image, faster startup)
- Node.js clustering: forks one worker per CPU core (up to 4)
- Connection pool: 30 reusable PostgreSQL connections shared across workers

**Frontend (`frontend/Dockerfile`)**
- Multi-stage build: `npm run build` produces optimized static assets
- Served by a lightweight nginx container (not Vite)
- Static assets cached with immutable headers (Vite filename hashing)

**Host Nginx (`nginx/host-production.conf`)**
- SSL termination + HTTP→HTTPS redirect (via certbot on host)
- Rate limiting on API routes (10 req/s per IP, burst 30)
- Proxy buffering to prevent 502s during slow responses
- Extended timeouts for AI endpoints (180s for investment/health-score calls)
- Routes `/api/*` → backend:3000, `/` → frontend:3001

**PostgreSQL**
- `max_connections=200` (up from default 100)
- `shared_buffers=256MB`, `effective_cache_size=512MB`
- Tuned checkpoint, WAL, and memory settings

### Capacity guidelines

With the production stack on a 2-core / 4GB server:

| Metric | Expected capacity |
|--------|-------------------|
| Concurrent users | 50–100 |
| API requests/sec | ~200 |
| DB connections | 30 per backend worker × workers |
| Frontend serving | Static files, effectively unlimited |

For higher loads, scale the backend horizontally with Docker Swarm or
Kubernetes replicas.

---

## SSL with Certbot (Let's Encrypt)

SSL is handled entirely at the host level using certbot with the host nginx.
No Docker containers are involved in SSL termination.

### Prerequisites

- A public hostname with DNS pointing to this server
- Ports 80 and 443 open in the firewall
- Host nginx installed: `sudo apt install nginx` (Ubuntu/Debian)
- Certbot installed: `sudo apt install certbot python3-certbot-nginx`

### Obtain a certificate

If you followed the "Host nginx setup" section above, certbot was already
run as part of that process. If not:

```bash
# Ensure the host nginx config is in place first
sudo certbot --nginx -d YOUR_HOSTNAME
```

Certbot will:
1. Verify domain ownership via an ACME challenge on port 80
2. Obtain the certificate from Let's Encrypt
3. Automatically modify the nginx config to enable SSL
4. Set up an HTTP → HTTPS redirect

### Verify HTTPS

```bash
# Should return 200 with SSL
curl -I https://YOUR_HOSTNAME

# Should return 301 redirect to HTTPS
curl -I http://YOUR_HOSTNAME
```

### Auto-renewal

Certbot installs a systemd timer (or cron job) that checks for renewal
twice daily. Verify it's active:

```bash
sudo systemctl status certbot.timer
```

To test renewal without actually renewing:

```bash
sudo certbot renew --dry-run
```

Certbot automatically reloads nginx after a successful renewal.

---

## Backup the Local Test Database

### Full database dump (recommended)

From your **local development machine** where the app is currently running:

```bash
cd /path/to/HOA_Financial_Platform

# Dump the entire database (all schemas, roles, data)
docker compose exec -T postgres pg_dump \
  -U hoafinance \
  -d hoafinance \
  --no-owner \
  --no-privileges \
  --format=custom \
  -f /tmp/hoafinance_backup.dump

# Copy the dump file out of the container
docker compose cp postgres:/tmp/hoafinance_backup.dump ./hoafinance_backup.dump
```

The `--format=custom` flag produces a compressed binary format that supports
selective restore. The file is typically 50–80% smaller than plain SQL.

### Alternative: Plain SQL dump

If you prefer a human-readable SQL file:

```bash
docker compose exec -T postgres pg_dump \
  -U hoafinance \
  -d hoafinance \
  --no-owner \
  --no-privileges \
  > hoafinance_backup.sql
```

### Backup a single tenant schema

To export just one tenant (e.g., Pine Creek HOA):

```bash
docker compose exec -T postgres pg_dump \
  -U hoafinance \
  -d hoafinance \
  --no-owner \
  --no-privileges \
  --schema=tenant_pine_creek_hoa_q33i \
  > pine_creek_backup.sql
```

> **Finding a tenant's schema name:**
> ```bash
> docker compose exec -T postgres psql -U hoafinance -d hoafinance \
>   -c "SELECT name, schema_name FROM shared.organizations WHERE status = 'active';"
> ```

---

## Restore a Backup into the Staged Environment

### 1. Transfer the backup to the staging server

```bash
scp hoafinance_backup.dump user@staging-server:/opt/hoa-ledgeriq/
```

### 2. Ensure the stack is running

```bash
cd /opt/hoa-ledgeriq
docker compose up -d
```

### 3. Drop and recreate the database (clean slate)

```bash
# Connect to postgres and reset the database
docker compose exec -T postgres psql -U hoafinance -d postgres -c "
  SELECT pg_terminate_backend(pid)
  FROM pg_stat_activity
  WHERE datname = 'hoafinance' AND pid <> pg_backend_pid();
"
docker compose exec -T postgres dropdb -U hoafinance hoafinance
docker compose exec -T postgres createdb -U hoafinance hoafinance
```

### 4a. Restore from custom-format dump

```bash
# Copy the dump into the container
docker compose cp hoafinance_backup.dump postgres:/tmp/hoafinance_backup.dump

# Restore
docker compose exec -T postgres pg_restore \
  -U hoafinance \
  -d hoafinance \
  --no-owner \
  --no-privileges \
  /tmp/hoafinance_backup.dump
```

### 4b. Restore from plain SQL dump

```bash
docker compose exec -T postgres psql \
  -U hoafinance \
  -d hoafinance \
  < hoafinance_backup.sql
```

### 5. Restart the backend

After restoring, restart the backend so NestJS re-establishes its connection
pool and picks up the restored schemas:

```bash
docker compose restart backend
```

---

## Running Migrations on the Staged Environment

Migrations live in `db/migrations/` and are numbered sequentially. After
restoring an older backup, you may need to apply newer migrations.

Check which migrations exist:

```bash
ls -la db/migrations/
```

Apply them in order:

```bash
# Run all migrations sequentially
for f in db/migrations/*.sql; do
  echo "Applying $f ..."
  docker compose exec -T postgres psql \
    -U hoafinance \
    -d hoafinance \
    < "$f"
done
```

Or apply a specific migration:

```bash
docker compose exec -T postgres psql \
  -U hoafinance \
  -d hoafinance \
  < db/migrations/010-health-scores.sql
```

> **Note:** Migrations are idempotent where possible (`IF NOT EXISTS`,
> `DO $$ ... $$` blocks), so re-running one that has already been applied
> is generally safe.

---

## Verifying the Deployment

### Quick health checks

```bash
# Backend is responding
curl -s http://localhost:3000/api/auth/login | head -c 100

# Database is accessible
docker compose exec -T postgres psql -U hoafinance -d hoafinance \
  -c "SELECT count(*) AS tenants FROM shared.organizations WHERE status = 'active';"

# Redis is working
docker compose exec -T redis redis-cli ping
```

### Full smoke test

1. Open `https://YOUR_HOSTNAME` (or `http://<server-ip>`) in a browser
2. Log in with a known account
3. Navigate to Dashboard — verify health scores load
4. Navigate to Capital Planning — verify Kanban columns render
5. Navigate to Projects — verify project list loads
6. Check the Settings page — version should read **2026.3.2 (beta)**

### Verify SSL (if enabled)

```bash
# Check certificate details
echo | openssl s_client -connect YOUR_HOSTNAME:443 -servername YOUR_HOSTNAME 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

# Check that HTTP redirects to HTTPS
curl -sI http://YOUR_HOSTNAME | grep -E 'HTTP|Location'
```

### View logs

```bash
docker compose logs -f              # all services
docker compose logs -f backend      # backend only
docker compose logs -f postgres     # database only
docker compose logs -f frontend     # frontend nginx
sudo tail -f /var/log/nginx/access.log  # host nginx access log
sudo tail -f /var/log/nginx/error.log   # host nginx error log
```

---

## Environment Variable Reference

| Variable          | Required | Description                                        |
|-------------------|----------|----------------------------------------------------|
| `POSTGRES_USER`   | Yes      | PostgreSQL username                                |
| `POSTGRES_PASSWORD`| Yes     | PostgreSQL password (**change from default**)      |
| `POSTGRES_DB`     | Yes      | Database name                                      |
| `DATABASE_URL`    | Yes      | Full connection string for the backend             |
| `REDIS_URL`       | Yes      | Redis connection string                            |
| `JWT_SECRET`      | Yes      | Secret for signing JWT tokens (**change from default**) |
| `NODE_ENV`        | Yes      | `development` or `production`                      |
| `AI_API_URL`      | Yes      | OpenAI-compatible inference endpoint               |
| `AI_API_KEY`      | Yes      | API key for AI provider (Nvidia)                   |
| `AI_MODEL`        | Yes      | Model identifier for AI calls                      |
| `AI_DEBUG`        | No       | Set `true` to log raw AI prompts/responses         |

---

## Architecture Overview

```
  Development:
                     ┌──────────────────┐
  Browser ─────────► │  nginx :80       │
                     └────────┬─────────┘
                   ┌──────────┴──────────┐
                   ▼                     ▼
          ┌──────────────┐       ┌──────────────┐
          │ backend :3000│       │frontend :5173│
          │   (NestJS)   │       │ (Vite/React) │
          └──────┬───────┘       └──────────────┘
            ┌────┴────┐
            ▼         ▼
    ┌────────────┐ ┌───────────┐
    │postgres:5432│ │redis :6379│
    │  (PG 15)   │ │ (Redis 7) │
    └────────────┘ └───────────┘

  Production (host nginx handles SSL + routing):
                     ┌────────────────────────────────┐
  Browser ─────────► │  Host nginx :80/:443 (SSL)     │
                     │  /api/* → 127.0.0.1:3000       │
                     │  /*     → 127.0.0.1:3001       │
                     └────────┬───────────┬───────────┘
                              ▼           ▼
                   ┌──────────────┐ ┌──────────────┐
                   │ backend :3000│ │frontend :3001│
                   │  (compiled)  │ │ (static nginx)│
                   └──────┬───────┘ └──────────────┘
                     ┌────┴────┐
                     ▼         ▼
             ┌────────────┐ ┌───────────┐
             │postgres:5432│ │redis :6379│
             │  (PG 15)   │ │ (Redis 7) │
             └────────────┘ └───────────┘
```

**Multi-tenant isolation:** Each HOA organization gets its own PostgreSQL
schema (e.g., `tenant_pine_creek_hoa_q33i`). The `shared` schema holds
cross-tenant tables (users, organizations, market rates). Tenant context
is resolved from the JWT token on every API request.