fix: health check now probes HTTP directly with 3-min timeout

The previous approach relied on Docker's container health status, but
Docker's healthcheck (start_period:30s + 3x15s retries = ~75s) marks
the container "unhealthy" before NestJS finishes cold-starting after a
fresh image build (New Relic + TypeORM + Redis + BullMQ init can take
2-3 minutes).

Changes:
- Primary check is now direct wget to localhost:3000/api from the host
- Docker health status used only for informational logging
- Total timeout increased from 130s to 190s (~3 min) for cold starts
- Early exit if container has stopped/exited (no point waiting)
- More backend log lines (30 vs 20) shown on failure for diagnostics

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

docs/gitea-runner-setup.md

@@ -0,0 +1,230 @@
+# Gitea Actions Runner Setup — HOALedgerIQ Production Server
+
+This guide walks through setting up a self-hosted Gitea Actions runner on the production server so the deployment workflow (`.gitea/workflows/deploy.yml`) can execute automatically.
+
+The runner uses **host execution mode** — jobs run directly on the server (not inside Docker containers) so the deploy script has access to Docker, the git repo, and the local filesystem.
+
+---
+
+## Prerequisites
+
+- Ubuntu Linux production server
+- Gitea instance (e.g., `https://git.sensetostyle.com`)
+- Docker and Docker Compose installed on the server
+- The HOALedgerIQ repo cloned at `/opt/hoa-ledgeriq`
+
+---
+
+## Step 1: Enable Actions in Gitea
+
+Ensure Actions are enabled in your Gitea configuration (`/etc/gitea/app.ini`):
+
+```ini
+[actions]
+ENABLED = true
+```
+
+Restart Gitea after making changes:
+
+```bash
+sudo systemctl restart gitea
+```
+
+---
+
+## Step 2: Get a Registration Token
+
+1. Log into your Gitea instance
+2. Navigate to **Site Administration** → **Actions** → **Runners**
+3. Copy the **Registration Token**
+
+> **Tip:** For tighter security, you can get a repo-scoped token instead:
+> Repo → **Settings** → **Actions** → **Runners** → copy the token shown there.
+> This limits the runner to only execute workflows from that specific repository.
+
+---
+
+## Step 3: Install the Act Runner Binary
+
+```bash
+# Download the latest act_runner for x86_64 Linux
+wget https://dl.gitea.com/act_runner/latest/act_runner-linux-amd64
+
+# Make executable and install to system path
+chmod +x act_runner-linux-amd64
+sudo mv act_runner-linux-amd64 /usr/local/bin/act_runner
+
+# Verify installation
+act_runner --version
+```
+
+> For ARM64 servers, use `act_runner-linux-arm64` instead.
+
+---
+
+## Step 4: Generate and Edit the Configuration
+
+```bash
+sudo mkdir -p /etc/act_runner
+act_runner generate-config > /tmp/config.yaml
+```
+
+Edit `/tmp/config.yaml` and set the **labels to use host execution mode**:
+
+```yaml
+runner:
+  labels:
+    - "ubuntu-latest:host"
+    - "ubuntu-22.04:host"
+```
+
+The `:host` suffix tells the runner to execute jobs directly on the server rather than spinning up Docker containers. This is required because the deploy script needs access to:
+
+- The Docker socket (to run `docker compose`)
+- The git repository at `/opt/hoa-ledgeriq`
+- The backup scripts and database
+
+Move the config into place and lock down permissions:
+
+```bash
+sudo mv /tmp/config.yaml /etc/act_runner/config.yaml
+sudo chmod 600 /etc/act_runner/config.yaml
+```
+
+---
+
+## Step 5: Register the Runner
+
+```bash
+act_runner register \
+  --no-interactive \
+  --instance "https://git.sensetostyle.com" \
+  --token "YOUR_REGISTRATION_TOKEN_HERE" \
+  --name "hoaledgeriq-prod" \
+  --labels "ubuntu-latest:host,ubuntu-22.04:host" \
+  --config /etc/act_runner/config.yaml
+```
+
+This creates a `.runner` file in the current directory containing the registration state.
+
+> **Interactive alternative:** Run `act_runner register --config /etc/act_runner/config.yaml` and follow the prompts.
+
+---
+
+## Step 6: Set Up as a Systemd Service
+
+Create the service file at `/etc/systemd/system/act_runner.service`:
+
+```ini
+[Unit]
+Description=Gitea Actions Runner (HOALedgerIQ Prod)
+Documentation=https://docs.gitea.com/usage/actions/act-runner
+After=docker.service network-online.target
+
+[Service]
+Type=simple
+User=root
+WorkingDirectory=/opt/hoa-ledgeriq
+ExecStart=/usr/local/bin/act_runner daemon --config /etc/act_runner/config.yaml
+Restart=always
+RestartSec=10
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=multi-user.target
+```
+
+> **Security note on `User=root`:** The deploy script needs to run `docker compose`, `git reset --hard`, etc. If you have a dedicated deploy user in the `docker` group with write access to `/opt/hoa-ledgeriq`, use that instead. Running as root is the simplest option but grants maximum privileges.
+
+Enable and start the service:
+
+```bash
+sudo systemctl daemon-reload
+sudo systemctl enable act_runner
+sudo systemctl start act_runner
+```
+
+---
+
+## Step 7: Verify the Runner Is Online
+
+Check the service is running:
+
+```bash
+sudo systemctl status act_runner
+```
+
+View logs:
+
+```bash
+sudo journalctl -u act_runner -f
+```
+
+Then confirm in Gitea:
+
+1. Go to **Site Administration** → **Actions** → **Runners**
+2. You should see **"hoaledgeriq-prod"** listed with status **Online**
+
+---
+
+## Step 8: Test the Workflow
+
+1. Go to your repo on Gitea → **Actions** tab
+2. Select the **"Deploy to Production"** workflow
+3. Click **Run Workflow**
+4. If this is the first deployment against an existing database, check the **"Mark existing migrations as applied"** box
+5. Monitor the run in the Actions tab
+
+---
+
+## Troubleshooting
+
+### Runner shows as Offline
+
+```bash
+# Check service status and logs
+sudo systemctl status act_runner
+sudo journalctl -u act_runner -n 50
+
+# Verify the instance URL is reachable from the server
+wget -qO- https://git.sensetostyle.com/api/v1/version
+```
+
+### Workflow stuck on "Waiting for runner"
+
+- Verify the runner labels match what the workflow expects. The workflow uses `runs-on: ubuntu-latest` which must match the `ubuntu-latest:host` label.
+- Check the runner is registered at the correct scope (instance-wide, org-level, or repo-level).
+
+### Permission denied errors during deploy
+
+- Ensure the systemd service `User` has Docker access (`usermod -aG docker <user>`)
+- Ensure the user has write access to `/opt/hoa-ledgeriq`
+
+### Re-registering after token expiry
+
+```bash
+sudo systemctl stop act_runner
+# Get a new token from Gitea admin panel, then:
+act_runner register \
+  --no-interactive \
+  --instance "https://git.sensetostyle.com" \
+  --token "NEW_TOKEN_HERE" \
+  --name "hoaledgeriq-prod" \
+  --labels "ubuntu-latest:host,ubuntu-22.04:host" \
+  --config /etc/act_runner/config.yaml
+sudo systemctl start act_runner
+```
+
+---
+
+## Security Best Practices
+
+| Concern | Recommendation |
+|---------|----------------|
+| Runner user | Use a dedicated user with `docker` group access rather than `root` when possible |
+| Registration token | Rotate periodically in the Gitea admin panel |
+| Config file | Keep `/etc/act_runner/config.yaml` at mode `600` (owner-read only) |
+| Runner scope | Register at the **repo level** instead of instance-wide so only this repo can trigger deployments |
+| Workflow triggers | The deploy workflow uses `workflow_dispatch` (manual only) — no automatic triggers on push |
+| Network | Ensure Gitea is accessed over HTTPS with valid SSL certificates |

scripts/deploy-prod.sh

@@ -40,9 +40,9 @@ DB_USER="${POSTGRES_USER:-hoafinance}"
 DB_NAME="${POSTGRES_DB:-hoafinance}"
 MIGRATION_DIR="$PROJECT_DIR/db/migrations"
 HEALTH_URL="http://localhost:3000/api"
-HEALTH_RETRIES=20
+HEALTH_RETRIES=36
 HEALTH_INTERVAL=5
-HEALTH_START_WAIT=30
+HEALTH_START_WAIT=10
 LOG_DIR="$PROJECT_DIR/logs"
 LOG_FILE="$LOG_DIR/deploy-$(date +%Y%m%d_%H%M%S).log"
 
@@ -359,34 +359,51 @@ fi
 # ====================================================================
 echo ""
 log "--- Step 5/6: Verifying application health ---"
-log "Waiting ${HEALTH_START_WAIT}s for backend to initialize ..."
+
+# After a fresh image build, NestJS cold-start can take 2-3 minutes:
+#   New Relic init → TypeORM connections → Redis → BullMQ → NestJS bootstrap
+# Docker's own healthcheck (start_period:30s + 3×15s retries = ~75s) is too
+# aggressive and will mark the container "unhealthy" before the app finishes
+# booting. So we do NOT rely on Docker's health status — we probe the HTTP
+# endpoint directly from the host and give it up to ~3 minutes total.
+TOTAL_WAIT=$((HEALTH_START_WAIT + HEALTH_RETRIES * HEALTH_INTERVAL))
+log "Will wait up to ${TOTAL_WAIT}s for backend to respond at $HEALTH_URL ..."
 sleep "$HEALTH_START_WAIT"
 
 HEALTHY=false
 for ((i=1; i<=HEALTH_RETRIES; i++)); do
-  if curl -sf "$HEALTH_URL" >/dev/null 2>&1; then
+  # Direct HTTP check from the host using wget (available on Ubuntu)
+  if wget -qO- --timeout=5 "$HEALTH_URL" >/dev/null 2>&1; then
     HEALTHY=true
     break
   fi
-  log "  Health check attempt $i/$HEALTH_RETRIES failed, retrying in ${HEALTH_INTERVAL}s ..."
+
+  # Also check Docker's container health for informational logging
+  CONTAINER_HEALTH=$($COMPOSE_CMD ps backend --format '{{.Health}}' 2>/dev/null || echo "unknown")
+
+  # If the container exited or was removed, fail immediately — no point waiting
+  CONTAINER_STATUS=$($COMPOSE_CMD ps backend --format '{{.Status}}' 2>/dev/null || echo "unknown")
+  if echo "$CONTAINER_STATUS" | grep -qi "exit\|dead\|removed"; then
+    err "Backend container has stopped unexpectedly: $CONTAINER_STATUS"
+    break
+  fi
+
+  log "  Health check attempt $i/$HEALTH_RETRIES — docker: ${CONTAINER_HEALTH}, retrying in ${HEALTH_INTERVAL}s ..."
   sleep "$HEALTH_INTERVAL"
 done
 
 if [ "$HEALTHY" = true ]; then
   ok "Backend is healthy and responding at $HEALTH_URL"
 else
-  err "Backend failed to respond after $((HEALTH_START_WAIT + HEALTH_RETRIES * HEALTH_INTERVAL))s"
+  # Log diagnostics before triggering rollback
+  err "Backend failed to respond after ${TOTAL_WAIT}s"
+  warn "Container status: $($COMPOSE_CMD ps backend 2>/dev/null || echo 'unknown')"
+  warn "Recent backend logs:"
+  $COMPOSE_CMD logs --tail=30 backend 2>/dev/null || true
   err "Triggering automatic rollback ..."
   exit 1  # trap will handle rollback
 fi
 
-# Also verify the container reports healthy via Docker
-if $COMPOSE_CMD ps backend 2>/dev/null | grep -q "healthy"; then
-  ok "Backend container health check: healthy"
-else
-  warn "Backend container health status is not 'healthy' yet (may still be within start_period)"
-fi
-
 # ====================================================================
 #  STEP 7: Post-upgrade database backup
 # ====================================================================