STR_Optimization_Manager/str-optimization-manager-spec.md

STR Optimization Manager — Claude Code Requirements Specification

1. Project Overview

A self-hosted, Dockerized web application for managing and optimizing a short-term rental (STR) property listed on multiple platforms (initially Airbnb and VRBO). The system automates daily performance data collection via browser automation, stores historical metrics in a local database, enables bulk pricing management across platforms, tracks pricing experiments (A/B style), and syncs reservation data for long-term record keeping.

Design philosophy: Modular by platform and by property. Every platform integration is an isolated adapter. Adding a new platform or a second property should require only a new adapter and config entry, not architectural changes.

---

2. Goals & Non-Goals

Goals

• Automated daily (and on-demand) scraping of performance metrics from Airbnb and VRBO
• Local time-series database of all collected metrics
• Dashboard with filterable, date-ranged charts for performance analysis
• Bulk pricing management across platforms (with preview/diff before commit)
• Pricing change log with experiment tagging and correlation to booking outcomes
• Full reservation sync and local storage
• Weekly performance summary email report
• Docker Compose deployment, runs on Mac (dev) and Debian (prod)
• Responsive UI: desktop and mobile

Non-Goals (explicitly out of scope for v1)

• Multi-property support (architecture should allow it later, but not built now)
• Cleaning/turnover scheduling
• Guest messaging
• Expense tracking / P&L
• Tax reporting
• Public platform APIs (all data collection is via authenticated browser sessions)

---

3. Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                        Docker Compose                        │
│                                                             │
│  ┌──────────────┐   ┌──────────────┐   ┌────────────────┐  │
│  │   Frontend   │   │  API Server  │   │ Scraper Worker │  │
│  │  (React/TS)  │◄──│ (Node/TS +   │◄──│  (Playwright + │  │
│  │   Vite SPA   │   │  Fastify)    │   │   adapters)    │  │
│  └──────────────┘   └──────┬───────┘   └───────┬────────┘  │
│                            │                    │           │
│                    ┌───────▼────────────────────▼───────┐   │
│                    │         PostgreSQL Database         │   │
│                    └────────────────────────────────────┘   │
│                                                             │
│  ┌──────────────────────────────────────────────────────┐   │
│  │              Scheduler (node-cron in API)             │   │
│  └──────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘

Services

Service	Image	Purpose
frontend	Node (build) → nginx	React SPA served via nginx
api	Node 20 Alpine	Fastify REST API + cron scheduler
scraper	Node 20 + Playwright	Browser automation workers
db	PostgreSQL 16 Alpine	Primary data store

---

4. Tech Stack

Layer	Technology
Backend API	Node.js 20 + TypeScript + Fastify
Browser Automation	Playwright (headless Chromium)
Database	PostgreSQL 16
ORM / Migrations	Drizzle ORM + drizzle-kit
Job Scheduling	node-cron
Frontend	React 18 + TypeScript + Vite
Charting	Recharts
UI Components	shadcn/ui + Tailwind CSS
Email	Nodemailer (SMTP)
Auth (app login)	Simple single-user session auth with bcrypt + JWT stored in httpOnly cookie
Container	Docker + Docker Compose v2
Secrets	.env file (never committed), credentials encrypted at rest in DB using AES-256

---

5. Platform Adapter Interface

Every platform integration implements this TypeScript interface. This is the core abstraction that keeps the system extensible.

interface PlatformAdapter {
  readonly platformId: string;          // e.g. 'airbnb' | 'vrbo'
  readonly displayName: string;

  // Session management
  login(credentials: Credentials): Promise<void>;
  isSessionValid(): Promise<boolean>;
  saveSession(store: SessionStore): Promise<void>;
  restoreSession(store: SessionStore): Promise<boolean>;

  // Data collection
  scrapePerformanceMetrics(): Promise<PerformanceSnapshot>;
  scrapeReservations(): Promise<Reservation[]>;
  scrapePricing(dateRange: DateRange): Promise<DailyPrice[]>;

  // Pricing mutations
  previewPriceChanges(changes: PriceChange[]): Promise<PriceChangeDiff>;
  applyPriceChanges(changes: PriceChange[]): Promise<PriceChangeResult>;

  // Adapter health
  selfTest(): Promise<AdapterHealthStatus>;
}

Initial Adapters

• AirbnbAdapter — targets Airbnb Host dashboard
• VrboAdapter — targets VRBO Owner dashboard

Adapter File Structure

src/
  adapters/
    base/
      PlatformAdapter.ts         ← interface + shared types
      SessionStore.ts            ← encrypted session persistence
      AdapterRegistry.ts         ← registers all adapters
    airbnb/
      AirbnbAdapter.ts           ← main adapter class
      airbnb.selectors.ts        ← ALL CSS selectors isolated here
      airbnb.flows.ts            ← login, nav, scraping flows
    vrbo/
      VrboAdapter.ts
      vrbo.selectors.ts
      vrbo.flows.ts

Critical pattern: All CSS selectors and XPaths live in *.selectors.ts files only. When a platform updates their UI, only that file needs updating — never the business logic.

---

6. Database Schema

platforms

Column	Type	Notes
id	varchar PK	e.g. 'airbnb'
display_name	varchar
credentials_encrypted	text	AES-256 encrypted JSON
session_data_encrypted	text	Stored browser session
last_scrape_at	timestamptz
is_active	boolean

performance_snapshots

Column	Type	Notes
id	uuid PK
platform_id	varchar FK
captured_at	timestamptz	When this row was written
period_label	varchar	e.g. 'last_30_days'
views_search	integer	Times appeared in search
views_listing	integer	Times listing was clicked
conversion_rate	numeric	views_listing / views_search
bookings_count	integer
occupancy_rate	numeric	% of available days booked
avg_daily_rate	numeric
revenue_total	numeric
raw_json	jsonb	Full raw payload for future parsing

daily_prices

Column	Type	Notes
id	uuid PK
platform_id	varchar FK
date	date	The night in question
price	numeric
is_available	boolean
min_stay_nights	integer
synced_at	timestamptz

price_changes

Column	Type	Notes
id	uuid PK
platform_id	varchar FK
date	date	Night being changed
price_before	numeric
price_after	numeric
changed_at	timestamptz
changed_by	varchar	'scheduled' or 'manual'
note	text	User-provided reason
experiment_id	uuid FK nullable

experiments

Column	Type	Notes
id	uuid PK
name	varchar	e.g. "Lower weekend rate Jan test"
hypothesis	text	What you expect to happen
start_date	date
end_date	date
status	varchar	'active' | 'completed' | 'cancelled'
created_at	timestamptz
conclusion	text	Notes written at end

reservations

Column	Type	Notes
id	uuid PK
platform_id	varchar FK
platform_reservation_id	varchar	Native ID from platform
guest_name	varchar
check_in	date
check_out	date
nights	integer
guests_count	integer
nightly_rate	numeric
cleaning_fee	numeric
platform_fee	numeric
total_payout	numeric
status	varchar	'confirmed' | 'cancelled' | 'completed'
booked_at	timestamptz
synced_at	timestamptz
raw_json	jsonb

scrape_jobs

Column	Type	Notes
id	uuid PK
platform_id	varchar FK
job_type	varchar	'performance' | 'pricing' | 'reservations'
triggered_by	varchar	'schedule' | 'manual'
status	varchar	'pending' | 'running' | 'success' | 'failed'
started_at	timestamptz
completed_at	timestamptz
error_message	text
rows_collected	integer

---

7. API Endpoints

All endpoints are prefixed /api/v1. Auth required on all except /api/v1/auth/login.

Auth

Method	Path	Description
POST	/auth/login	App login (single user)
POST	/auth/logout	Clear session
GET	/auth/me	Current session info

Platforms

Method	Path	Description
GET	/platforms	List platforms and status
PUT	/platforms/:id/credentials	Update stored credentials
POST	/platforms/:id/test	Test login + adapter health
POST	/platforms/:id/scrape	Trigger on-demand scrape (all types)
GET	/platforms/:id/scrape-jobs	Recent job history

Performance

Method	Path	Description
GET	/performance/snapshots	Query snapshots, supports ?platform=&from=&to=
GET	/performance/summary	Aggregated summary across platforms
GET	/performance/trends	Time-series data for charts

Pricing

Method	Path	Description
GET	/pricing/calendar	All daily prices ?platform=&from=&to=
POST	/pricing/preview	Dry-run bulk price changes, returns diff
POST	/pricing/apply	Apply previewed changes after confirmation
GET	/pricing/changes	Price change log ?platform=&from=&to=&experiment_id=

Experiments

Method	Path	Description
GET	/experiments	List all experiments
POST	/experiments	Create new experiment
PUT	/experiments/:id	Update (add conclusion, change status)
GET	/experiments/:id/analysis	Correlation: price changes → bookings/views

Reservations

Method	Path	Description
GET	/reservations	List reservations ?platform=&status=&from=&to=
GET	/reservations/summary	Occupancy, revenue totals by month/year

Reports

Method	Path	Description
POST	/reports/weekly/send	Manually trigger weekly email report
GET	/reports/weekly/preview	Preview this week's report as JSON

---

8. Frontend — Pages & Views

Navigation Structure

/ (Dashboard)
/performance
/pricing
/experiments
/reservations
/settings

Dashboard (/)

• KPI cards: occupancy rate, avg daily rate, total revenue MTD, search views (last 30d)
• Side-by-side platform comparison for all KPIs
• Booking trend sparkline (last 90 days)
• Recent reservations list (last 5)
• Scraper job status indicators (last run time per platform, success/fail badge)
• "Run Scrape Now" button per platform

Performance (/performance)

• Date range picker (presets: 7d, 30d, 90d, YTD, custom)
• Platform filter toggle (All / Airbnb / VRBO)
• Charts (all use Recharts):
  • Search views over time (line)
  • Listing click-through rate over time (line)
  • Bookings per week (bar)
  • Occupancy rate over time (area)
  • Avg daily rate over time (line, overlaid with booking events)
• Data table below charts: raw snapshot history, exportable to CSV

Pricing (/pricing)

• Calendar grid view: each day shows price per platform, color-coded by deviation from base rate
• Sidebar panel: select date range + enter new price → generates preview
• Preview modal: shows diff table (date | platform | old price | new price) before any changes go live. Requires explicit "Confirm & Apply" button.
• Pricing change log table: filterable by platform, date range, experiment
• "Link to Experiment" action on any change or group of changes

Experiments (/experiments)

• List view: all experiments with status badge, date range, linked price changes count
• Create experiment modal: name, hypothesis, date range, initial notes
• Experiment detail page:
  • Linked price changes table
  • Performance chart for the experiment date range (views, bookings, occupancy)
  • Before/after comparison: avg metrics N days before vs during experiment
  • Conclusion text field (editable when status = completed)

Reservations (/reservations)

• Table: all reservations, sortable, filterable by platform/status/date range
• Monthly occupancy heatmap calendar
• Revenue by month bar chart
• YoY comparison once data spans 12+ months

Settings (/settings)

• Platform credentials (masked, update form per platform)
• App login password change
• Scrape schedule configuration (time of day for daily run)
• SMTP configuration for weekly report email
• Adapter health check panel: "Test Connection" per platform with live output log

---

9. Scraper Worker — Detailed Behavior

Session Management

• On first run, performs full login flow (email → password → handle any MFA prompt interactively via a special "needs attention" UI state)
• After successful login, saves browser storage state (cookies + localStorage) encrypted to DB
• On subsequent runs, restores session state and verifies validity before scraping
• If session invalid, re-triggers login flow and flags for user attention if MFA required

Scrape Job Flow

1. Job queued (by scheduler or API trigger)
2. Worker picks up job
3. Restore session → validate → re-login if needed
4. Navigate to performance dashboard → extract metrics → insert performance_snapshot row
5. Navigate to calendar/pricing → extract N days of pricing → upsert daily_prices rows
6. Navigate to reservations → extract all reservations → upsert reservations rows
7. Update scrape_jobs row with status + counts
8. Emit websocket event → UI updates in real time

Error Handling

• Retry up to 3 times on transient failures (network, selector not found)
• On persistent failure: mark job as failed, store error message, surface in UI
• Never silently swallow errors — all failures logged to scrape_jobs table
• Screenshot on failure: save to /data/screenshots/ volume for debugging

Anti-Detection Considerations (document in spec, implement in adapters)

• Randomized delays between actions (200–800ms)
• Human-like mouse movement patterns via Playwright's mouse.move()
• Persist and reuse sessions to minimize login frequency
• Run during off-peak hours by default (configurable)
• User-agent set to current stable Chrome

---

10. Pricing Change Flow (Safety-First)

This flow must never apply changes without explicit user confirmation.

User selects dates + enters new price
        ↓
POST /pricing/preview
        ↓
System queries current prices from daily_prices table
        ↓
Returns diff: [{date, platform, currentPrice, newPrice, delta}]
        ↓
UI renders preview modal with full diff table
        ↓
User reviews → clicks "Confirm & Apply"
        ↓
POST /pricing/apply (idempotency key from preview response)
        ↓
Scraper worker opens browser → navigates to platform calendar
        ↓
Applies changes date by date (with verification reads after each)
        ↓
Writes price_changes rows for each date changed
        ↓
Returns result: {success: [], failed: []}
        ↓
UI shows success/failure summary

---

11. Weekly Report Email

Sent every Monday at 8am (configurable). Contains:

• This week vs last week: views, clicks, CTR, bookings
• MTD vs same period last month: revenue, occupancy
• Upcoming 30 days: occupancy %, revenue booked
• Active experiments: name, days running, early metric movement
• Pricing changes this week: count, avg delta
• Any scraper failures from the past 7 days

Format: HTML email with inline styles (no external CSS). Plain-text fallback included.

---

12. Docker Compose — Full Stack Definition

# docker-compose.yml

version: '3.9'

services:
  db:
    image: postgres:16-alpine
    restart: unless-stopped
    environment:
      POSTGRES_DB: str_manager
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${DB_USER}"]
      interval: 10s
      timeout: 5s
      retries: 5

  api:
    build:
      context: ./apps/api
      dockerfile: Dockerfile
    restart: unless-stopped
    depends_on:
      db:
        condition: service_healthy
    environment:
      DATABASE_URL: postgresql://${DB_USER}:${DB_PASSWORD}@db:5432/str_manager
      JWT_SECRET: ${JWT_SECRET}
      ENCRYPTION_KEY: ${ENCRYPTION_KEY}
      SMTP_HOST: ${SMTP_HOST}
      SMTP_PORT: ${SMTP_PORT}
      SMTP_USER: ${SMTP_USER}
      SMTP_PASS: ${SMTP_PASS}
      REPORT_EMAIL_TO: ${REPORT_EMAIL_TO}
      SCRAPER_URL: http://scraper:3001
    ports:
      - "3000:3000"

  scraper:
    build:
      context: ./apps/scraper
      dockerfile: Dockerfile
    restart: unless-stopped
    depends_on:
      db:
        condition: service_healthy
    environment:
      DATABASE_URL: postgresql://${DB_USER}:${DB_PASSWORD}@db:5432/str_manager
      ENCRYPTION_KEY: ${ENCRYPTION_KEY}
      PLAYWRIGHT_HEADLESS: "true"
    volumes:
      - scraper_screenshots:/app/screenshots
    shm_size: '2gb'  # Required for Chromium in Docker

  frontend:
    build:
      context: ./apps/frontend
      dockerfile: Dockerfile
    restart: unless-stopped
    depends_on:
      - api
    ports:
      - "80:80"
      - "443:443"

volumes:
  postgres_data:
  scraper_screenshots:

.env.example

# Database
DB_USER=str_manager
DB_PASSWORD=changeme_strong_password

# App Security
JWT_SECRET=changeme_64_char_random_string
ENCRYPTION_KEY=changeme_32_char_aes_key

# Email (weekly report)
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=you@gmail.com
SMTP_PASS=your_app_password
REPORT_EMAIL_TO=you@gmail.com

# App login
APP_USERNAME=admin
APP_PASSWORD_HASH=bcrypt_hash_of_your_password

---

13. Monorepo Structure

str-optimization-manager/
├── docker-compose.yml
├── docker-compose.dev.yml        ← mounts source for hot reload
├── .env.example
├── .gitignore                    ← must include .env, screenshots/
├── README.md
├── apps/
│   ├── api/
│   │   ├── Dockerfile
│   │   ├── package.json
│   │   ├── tsconfig.json
│   │   └── src/
│   │       ├── index.ts          ← Fastify server entry
│   │       ├── routes/           ← one file per route group
│   │       ├── services/         ← business logic
│   │       ├── db/
│   │       │   ├── schema.ts     ← Drizzle schema (single source of truth)
│   │       │   └── migrations/
│   │       ├── scheduler/
│   │       │   └── cron.ts
│   │       └── email/
│   │           └── weeklyReport.ts
│   ├── scraper/
│   │   ├── Dockerfile
│   │   ├── package.json
│   │   ├── tsconfig.json
│   │   └── src/
│   │       ├── index.ts          ← HTTP server (receives jobs from api)
│   │       ├── adapters/         ← see section 5
│   │       ├── queue/            ← in-memory job queue
│   │       └── utils/
│   │           ├── browser.ts    ← Playwright browser factory
│   │           └── encryption.ts ← AES-256 helpers
│   └── frontend/
│       ├── Dockerfile
│       ├── nginx.conf
│       ├── package.json
│       ├── vite.config.ts
│       └── src/
│           ├── main.tsx
│           ├── App.tsx
│           ├── pages/
│           ├── components/
│           ├── hooks/
│           └── lib/
│               ├── api.ts        ← typed API client
│               └── utils.ts
└── packages/
    └── shared-types/             ← shared TypeScript types between api/scraper/frontend
        └── src/
            └── index.ts

---

14. UI Design Direction

Aesthetic: Clean, data-dense, utilitarian dashboard. Think Vercel analytics meets a Bloomberg terminal — dark mode default, monospaced numbers, high contrast data visualizations. Not a generic SaaS template.

Color palette:

• Background: #0a0a0a (near-black)
• Surface: #141414
• Border: #262626
• Primary accent: #22c55e (green — for positive metrics, occupancy, revenue)
• Warning: #f59e0b
• Danger: #ef4444
• Text primary: #fafafa
• Text muted: #737373
• Chart colors: distinct accessible palette (no red/green only for accessibility)

Typography:

• Numbers/data: JetBrains Mono or IBM Plex Mono — monospaced for alignment
• UI labels: Geist or DM Sans — clean, modern, not generic

Interaction patterns:

• All data tables have column sorting
• All date range pickers have keyboard support
• Loading states: skeleton screens (not spinners)
• Real-time job status via WebSocket (SSE acceptable as simpler alternative)
• Mobile: bottom tab navigation, cards stack vertically, charts scroll horizontally

---

15. Key Implementation Notes for Claude Code

1. Start with the database schema and Drizzle migrations — everything else derives from this
2. Build the adapter interface and a mock adapter first — use the mock for all frontend/API development before real scrapers are needed
3. The Airbnb adapter is higher priority than VRBO — build and test it first
4. All selector strings must be constants — never inline a CSS selector in logic code
5. Preview-before-apply is non-negotiable — the /pricing/apply endpoint must reject requests without a valid preview token
6. Session encryption is day-one, not a later hardening step — credentials never touch disk unencrypted
7. The weekly email must work with any standard SMTP provider — no vendor lock-in (no SendGrid dependency)
8. Write a docker-compose.dev.yml that mounts source volumes and enables hot reload for both api and frontend
9. Include a /api/v1/health endpoint that checks DB connectivity and returns scraper worker status
10. The README must include first-run setup steps, how to update platform credentials, how to add a new platform adapter, and how to run outside Docker for local development

---

16. Acceptance Criteria

The following must all be true for v1 to be considered complete:

• docker compose up from a fresh clone brings the full stack online
• App login works (single user, password-protected)
• Airbnb credentials can be entered, tested, and stored encrypted
• VRBO credentials can be entered, tested, and stored encrypted
• Manual "scrape now" triggers all three data collection types per platform
• Daily cron scrape runs at configured time
• Performance dashboard renders with real data from at least one platform
• All performance charts filter correctly by platform and date range
• Pricing calendar shows current prices per platform per day
• Bulk price change goes through preview → confirm → apply flow with no way to skip preview
• Every applied price change is recorded in price_changes table
• An experiment can be created, price changes linked to it, and the analysis view shows before/after metric comparison
• All reservations sync and display in reservations table
• Weekly report email sends successfully via configured SMTP
• UI is usable on a 390px wide mobile screen
• Scraper failure is visible in the UI within 60 seconds of occurrence
• .env.example covers every required environment variable