myTradeTracker/RATE_LIMITING_SOLUTION.md

Rate Limiting & Caching Solution for Yahoo Finance API

Problem

Yahoo Finance API has rate limits and was returning HTTP 429 (Too Many Requests) errors when the dashboard loaded. The dashboard would:

1. Fetch prices for every open position synchronously
2. Block UI until all prices were loaded
3. Hit rate limits quickly with multiple open positions
4. Lose all cached data on container restart (in-memory cache only)

Solution Overview

Implemented a multi-layered approach:

1. Database-backed price cache - Persistent across restarts
2. Rate limiting with exponential backoff - Respects Yahoo Finance limits
3. Batch processing - Fetches multiple prices efficiently
4. Stale-while-revalidate pattern - UI shows cached data immediately
5. Background refresh - Optional manual price updates
6. Configurable API call limits - Control how many API calls to make

Architecture

New Components

1. MarketPrice Model (backend/app/models/market_price.py)

Database table to cache prices with timestamps:

- symbol: Stock ticker (indexed, unique)
- price: Current price
- fetched_at: When price was fetched
- source: Data source (yahoo_finance)

2. MarketDataService (backend/app/services/market_data_service.py)

Core service handling all market data:

Features:

• Database caching: Stores prices in PostgreSQL
• Rate limiting: 500ms delay between requests, exponentially backs off on 429 errors
• Retry logic: Up to 3 retries with increasing delays
• Batch fetching: get_prices_batch() fetches multiple symbols efficiently
• Stale data support: Returns old cached data if fresh fetch fails
• Background refresh: refresh_stale_prices() for periodic maintenance

Key Methods:

get_price(symbol, allow_stale=True)
# Returns cached price if fresh, or fetches from Yahoo

get_prices_batch(symbols, allow_stale=True, max_fetches=10)
# Fetches multiple symbols with rate limiting

refresh_stale_prices(min_age_seconds=300, limit=20)
# Background task to refresh old prices

3. PerformanceCalculatorV2 (backend/app/services/performance_calculator_v2.py)

Enhanced calculator using MarketDataService:

Features:

• Batch price fetching for all open positions
• Configurable API call limits
• Returns cache statistics
• Non-blocking operation

Key Changes:

calculate_account_stats(
    account_id,
    update_prices=True,  # Set to False to use only cache
    max_api_calls=10     # Limit Yahoo Finance API calls
)

4. Enhanced Analytics Endpoints (backend/app/api/endpoints/analytics_v2.py)

New/Updated Endpoints:

GET /api/analytics/overview/{account_id}?refresh_prices=false&max_api_calls=5
# Default: uses cached prices only (fast!)
# Set refresh_prices=true to fetch fresh data

POST /api/analytics/refresh-prices/{account_id}?max_api_calls=10
# Manual refresh - waits for completion

POST /api/analytics/refresh-prices-background/{account_id}?max_api_calls=20
# Background refresh - returns immediately

POST /api/analytics/refresh-stale-cache?min_age_minutes=10&limit=20
# Maintenance endpoint for periodic cache refresh

DELETE /api/analytics/clear-old-cache?older_than_days=30
# Clean up old cached prices

5. DashboardV2 Component (frontend/src/components/DashboardV2.tsx)

Features:

• Instant loading: Shows cached data immediately
• Data freshness indicator: Shows when data was last updated
• Manual refresh button: User can trigger fresh price fetch
• Cache statistics: Shows how many prices were cached vs fetched
• Background updates: Refetches on window focus
• Stale-while-revalidate: Keeps old data visible while fetching new

User Experience:

1. Dashboard loads instantly with cached prices
2. User sees "Last updated: 2m ago"
3. Click "Refresh Prices" to get fresh data
4. Background spinner shows refresh in progress
5. Data updates when refresh completes

How It Works

First Load (No Cache)

1. User opens dashboard
2. Frontend calls GET /api/analytics/overview/{id}?refresh_prices=false
3. Backend checks database cache - empty
4. Returns stats with unrealized_pnl = null for open positions
5. Dashboard shows data immediately (without prices)
6. User clicks "Refresh Prices"
7. Fetches first 10 symbols from Yahoo Finance
8. Caches results in database
9. Updates dashboard with fresh prices

Subsequent Loads (With Cache)

1. User opens dashboard
2. Frontend calls GET /api/analytics/overview/{id}?refresh_prices=false
3. Backend checks database cache - HIT!
4. Returns stats with cached prices (instant!)
5. Dashboard shows: "Last updated: 3m ago | 📦 8 cached"
6. User can optionally click "Refresh Prices" for fresh data

Background Refresh

1. Cron job calls POST /api/analytics/refresh-stale-cache
2. Finds prices older than 10 minutes
3. Refreshes up to 20 prices with rate limiting
4. Next dashboard load has fresher cache

Configuration

Backend Settings (backend/app/config.py)

MARKET_DATA_CACHE_TTL: int = 300  # 5 minutes (adjust as needed)

Frontend Settings (frontend/src/components/DashboardV2.tsx)

staleTime: 30000,  # Keep cache for 30 seconds
refetchOnWindowFocus: true,  # Auto-refresh when user returns

Per-Request Controls

// Fast load with cached data only
analyticsApi.getOverview(accountId, {
  refresh_prices: false,
  max_api_calls: 0
})

// Fresh data with limited API calls
analyticsApi.getOverview(accountId, {
  refresh_prices: true,
  max_api_calls: 10  // Fetch up to 10 symbols
})

Rate Limiting Strategy

The MarketDataService implements smart rate limiting:

1. Initial delay: 500ms between requests
2. Exponential backoff: Doubles delay on 429 errors (up to 10s max)
3. Gradual recovery: Decreases delay by 10% on successful requests
4. Retry logic: Up to 3 retries with increasing delays

Example flow:

Request 1: Success (500ms delay)
Request 2: Success (450ms delay)
Request 3: 429 Error (delay → 900ms)
Request 3 retry 1: 429 Error (delay → 1800ms)
Request 3 retry 2: Success (delay → 1620ms)
Request 4: Success (delay → 1458ms)
...gradually returns to 500ms

Database Migration

Run migration to add market_prices table:

docker compose exec backend alembic upgrade head

Deployment Steps

1. Transfer new files to server:

# On Mac
cd /Users/chris/Desktop/fidelity

# Backend files
scp backend/app/models/market_price.py pi@starship2:~/fidelity/backend/app/models/
scp backend/app/services/market_data_service.py pi@starship2:~/fidelity/backend/app/services/
scp backend/app/services/performance_calculator_v2.py pi@starship2:~/fidelity/backend/app/services/
scp backend/app/api/endpoints/analytics_v2.py pi@starship2:~/fidelity/backend/app/api/endpoints/
scp backend/alembic/versions/add_market_prices_table.py pi@starship2:~/fidelity/backend/alembic/versions/
scp backend/app/models/__init__.py pi@starship2:~/fidelity/backend/app/models/

# Frontend files
scp frontend/src/components/DashboardV2.tsx pi@starship2:~/fidelity/frontend/src/components/
scp frontend/src/api/client.ts pi@starship2:~/fidelity/frontend/src/api/

2. Update main.py to use new analytics router:

# backend/app/main.py
from app.api.endpoints import analytics_v2

app.include_router(
    analytics_v2.router,
    prefix=f"{settings.API_V1_PREFIX}/analytics",
    tags=["analytics"]
)

3. Update App.tsx to use DashboardV2:

// frontend/src/App.tsx
import DashboardV2 from './components/DashboardV2';

// Replace <Dashboard /> with <DashboardV2 />

4. Run migration and rebuild:

ssh pi@starship2
cd ~/fidelity

# Stop containers
docker compose down

# Rebuild
docker compose build --no-cache backend frontend

# Start
docker compose up -d

# Run migration
docker compose exec backend alembic upgrade head

# Verify table was created
docker compose exec postgres psql -U fidelity -d fidelitytracker -c "\d market_prices"

Testing

Test the improved dashboard:

# 1. Open dashboard - should load instantly with cached data
open http://starship2:3000

# 2. Check logs - should see cache HITs, not Yahoo Finance requests
docker compose logs backend | grep -i "cache\|yahoo"

# 3. Click "Refresh Prices" button
# Should see rate-limited requests in logs

# 4. Check database cache
docker compose exec postgres psql -U fidelity -d fidelitytracker -c "SELECT symbol, price, fetched_at FROM market_prices ORDER BY fetched_at DESC LIMIT 10;"

Test API endpoints directly:

# Fast load with cache only
curl "http://localhost:8000/api/analytics/overview/1?refresh_prices=false&max_api_calls=0"

# Fresh data with limited API calls
curl "http://localhost:8000/api/analytics/overview/1?refresh_prices=true&max_api_calls=5"

# Manual refresh
curl -X POST "http://localhost:8000/api/analytics/refresh-prices/1?max_api_calls=10"

# Background refresh (returns immediately)
curl -X POST "http://localhost:8000/api/analytics/refresh-prices-background/1?max_api_calls=15"

Benefits

Before:

• ❌ Dashboard blocked for 30+ seconds
• ❌ Hit rate limits constantly (429 errors)
• ❌ Lost all cache data on restart
• ❌ No way to control API usage
• ❌ Poor user experience

After:

• ✅ Dashboard loads instantly (<1 second)
• ✅ Respects rate limits with exponential backoff
• ✅ Persistent cache across restarts
• ✅ Configurable API call limits
• ✅ Shows stale data while refreshing
• ✅ Manual refresh option
• ✅ Background updates
• ✅ Cache statistics visible to user

Maintenance

Periodic cache refresh (optional):

# Add to crontab for periodic refresh
*/10 * * * * curl -X POST "http://localhost:8000/api/analytics/refresh-stale-cache?min_age_minutes=10&limit=20"

Clear old cache:

# Monthly cleanup
curl -X DELETE "http://localhost:8000/api/analytics/clear-old-cache?older_than_days=30"

Future Enhancements

1. WebSocket updates: Push price updates to frontend in real-time
2. Batch updates: Update all accounts' prices in background job
3. Multiple data sources: Fall back to alternative APIs if Yahoo fails
4. Historical caching: Store price history for charting
5. Smart refresh: Only refresh prices during market hours

Troubleshooting

Still getting 429 errors:

• Increase _rate_limit_delay in MarketDataService
• Decrease max_api_calls in API requests
• Use longer cache_ttl (e.g., 600 seconds = 10 minutes)

Dashboard shows old data:

• Check cache_ttl setting
• Click "Refresh Prices" button
• Check database: SELECT * FROM market_prices;

Prices not updating:

• Check backend logs for errors
• Verify migration ran: \d market_prices in postgres
• Check if symbols are valid (Yahoo Finance format)

Summary

This solution provides a production-ready approach to handling rate-limited APIs with:

• Fast, responsive UI
• Persistent caching
• Graceful degradation
• User control
• Clear feedback

Users get instant dashboard loads with cached data, and can optionally refresh for the latest prices without blocking the UI.