chris/myTradeTracker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial release v1.1.0

Browse Source 
- Complete MVP for tracking Fidelity brokerage account performance
- Transaction import from CSV with deduplication
- Automatic FIFO position tracking with options support
- Real-time P&L calculations with market data caching
- Dashboard with timeframe filtering (30/90/180 days, 1 year, YTD, all time)
- Docker-based deployment with PostgreSQL backend
- React/TypeScript frontend with TailwindCSS
- FastAPI backend with SQLAlchemy ORM

Features:
- Multi-account support
- Import via CSV upload or filesystem
- Open and closed position tracking
- Balance history charting
- Performance analytics and metrics
- Top trades analysis
- Responsive UI design

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Chris
2026-01-22 14:27:43 -05:00
commit eea4469095
90 changed files with 14513 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

363
RATE_LIMITING_SOLUTION.md Normal file
View File

@@ -0,0 +1,363 @@
### Rate Limiting & Caching Solution for Yahoo Finance API
## Problem
Yahoo Finance API has rate limits and was returning **HTTP 429 (Too Many Requests)** errors when the dashboard loaded. The dashboard would:
1. Fetch prices for every open position synchronously
2. Block UI until all prices were loaded
3. Hit rate limits quickly with multiple open positions
4. Lose all cached data on container restart (in-memory cache only)
## Solution Overview
Implemented a multi-layered approach:
1. **Database-backed price cache** - Persistent across restarts
2. **Rate limiting with exponential backoff** - Respects Yahoo Finance limits
3. **Batch processing** - Fetches multiple prices efficiently
4. **Stale-while-revalidate pattern** - UI shows cached data immediately
5. **Background refresh** - Optional manual price updates
6. **Configurable API call limits** - Control how many API calls to make
## Architecture
### New Components
#### 1. `MarketPrice` Model (`backend/app/models/market_price.py`)
Database table to cache prices with timestamps:
```python
- symbol: Stock ticker (indexed, unique)
- price: Current price
- fetched_at: When price was fetched
- source: Data source (yahoo_finance)
```
#### 2. `MarketDataService` (`backend/app/services/market_data_service.py`)
Core service handling all market data:
**Features:**
- **Database caching**: Stores prices in PostgreSQL
- **Rate limiting**: 500ms delay between requests, exponentially backs off on 429 errors
- **Retry logic**: Up to 3 retries with increasing delays
- **Batch fetching**: `get_prices_batch()` fetches multiple symbols efficiently
- **Stale data support**: Returns old cached data if fresh fetch fails
- **Background refresh**: `refresh_stale_prices()` for periodic maintenance
**Key Methods:**
```python
get_price(symbol, allow_stale=True)
# Returns cached price if fresh, or fetches from Yahoo
get_prices_batch(symbols, allow_stale=True, max_fetches=10)
# Fetches multiple symbols with rate limiting
refresh_stale_prices(min_age_seconds=300, limit=20)
# Background task to refresh old prices
```
#### 3. `PerformanceCalculatorV2` (`backend/app/services/performance_calculator_v2.py`)
Enhanced calculator using `MarketDataService`:
**Features:**
- Batch price fetching for all open positions
- Configurable API call limits
- Returns cache statistics
- Non-blocking operation
**Key Changes:**
```python
calculate_account_stats(
account_id,
update_prices=True, # Set to False to use only cache
max_api_calls=10 # Limit Yahoo Finance API calls
)
```
#### 4. Enhanced Analytics Endpoints (`backend/app/api/endpoints/analytics_v2.py`)
**New/Updated Endpoints:**
```
GET /api/analytics/overview/{account_id}?refresh_prices=false&max_api_calls=5
# Default: uses cached prices only (fast!)
# Set refresh_prices=true to fetch fresh data
POST /api/analytics/refresh-prices/{account_id}?max_api_calls=10
# Manual refresh - waits for completion
POST /api/analytics/refresh-prices-background/{account_id}?max_api_calls=20
# Background refresh - returns immediately
POST /api/analytics/refresh-stale-cache?min_age_minutes=10&limit=20
# Maintenance endpoint for periodic cache refresh
DELETE /api/analytics/clear-old-cache?older_than_days=30
# Clean up old cached prices
```
#### 5. `DashboardV2` Component (`frontend/src/components/DashboardV2.tsx`)
**Features:**
- **Instant loading**: Shows cached data immediately
- **Data freshness indicator**: Shows when data was last updated
- **Manual refresh button**: User can trigger fresh price fetch
- **Cache statistics**: Shows how many prices were cached vs fetched
- **Background updates**: Refetches on window focus
- **Stale-while-revalidate**: Keeps old data visible while fetching new
**User Experience:**
1. Dashboard loads instantly with cached prices
2. User sees "Last updated: 2m ago"
3. Click "Refresh Prices" to get fresh data
4. Background spinner shows refresh in progress
5. Data updates when refresh completes
## How It Works
### First Load (No Cache)
```
1. User opens dashboard
2. Frontend calls GET /api/analytics/overview/{id}?refresh_prices=false
3. Backend checks database cache - empty
4. Returns stats with unrealized_pnl = null for open positions
5. Dashboard shows data immediately (without prices)
6. User clicks "Refresh Prices"
7. Fetches first 10 symbols from Yahoo Finance
8. Caches results in database
9. Updates dashboard with fresh prices
```
### Subsequent Loads (With Cache)
```
1. User opens dashboard
2. Frontend calls GET /api/analytics/overview/{id}?refresh_prices=false
3. Backend checks database cache - HIT!
4. Returns stats with cached prices (instant!)
5. Dashboard shows: "Last updated: 3m ago | 📦 8 cached"
6. User can optionally click "Refresh Prices" for fresh data
```
### Background Refresh
```
1. Cron job calls POST /api/analytics/refresh-stale-cache
2. Finds prices older than 10 minutes
3. Refreshes up to 20 prices with rate limiting
4. Next dashboard load has fresher cache
```
## Configuration
### Backend Settings (`backend/app/config.py`)
```python
MARKET_DATA_CACHE_TTL: int = 300 # 5 minutes (adjust as needed)
```
### Frontend Settings (`frontend/src/components/DashboardV2.tsx`)
```typescript
staleTime: 30000, # Keep cache for 30 seconds
refetchOnWindowFocus: true, # Auto-refresh when user returns
```
### Per-Request Controls
```typescript
// Fast load with cached data only
analyticsApi.getOverview(accountId, {
refresh_prices: false,
max_api_calls: 0
})
// Fresh data with limited API calls
analyticsApi.getOverview(accountId, {
refresh_prices: true,
max_api_calls: 10 // Fetch up to 10 symbols
})
```
## Rate Limiting Strategy
The `MarketDataService` implements smart rate limiting:
1. **Initial delay**: 500ms between requests
2. **Exponential backoff**: Doubles delay on 429 errors (up to 10s max)
3. **Gradual recovery**: Decreases delay by 10% on successful requests
4. **Retry logic**: Up to 3 retries with increasing delays
Example flow:
```
Request 1: Success (500ms delay)
Request 2: Success (450ms delay)
Request 3: 429 Error (delay → 900ms)
Request 3 retry 1: 429 Error (delay → 1800ms)
Request 3 retry 2: Success (delay → 1620ms)
Request 4: Success (delay → 1458ms)
...gradually returns to 500ms
```
## Database Migration
Run migration to add market_prices table:
```bash
docker compose exec backend alembic upgrade head
```
## Deployment Steps
### 1. Transfer new files to server:
```bash
# On Mac
cd /Users/chris/Desktop/fidelity
# Backend files
scp backend/app/models/market_price.py pi@starship2:~/fidelity/backend/app/models/
scp backend/app/services/market_data_service.py pi@starship2:~/fidelity/backend/app/services/
scp backend/app/services/performance_calculator_v2.py pi@starship2:~/fidelity/backend/app/services/
scp backend/app/api/endpoints/analytics_v2.py pi@starship2:~/fidelity/backend/app/api/endpoints/
scp backend/alembic/versions/add_market_prices_table.py pi@starship2:~/fidelity/backend/alembic/versions/
scp backend/app/models/__init__.py pi@starship2:~/fidelity/backend/app/models/
# Frontend files
scp frontend/src/components/DashboardV2.tsx pi@starship2:~/fidelity/frontend/src/components/
scp frontend/src/api/client.ts pi@starship2:~/fidelity/frontend/src/api/
```
### 2. Update main.py to use new analytics router:
```python
# backend/app/main.py
from app.api.endpoints import analytics_v2
app.include_router(
analytics_v2.router,
prefix=f"{settings.API_V1_PREFIX}/analytics",
tags=["analytics"]
)
```
### 3. Update App.tsx to use DashboardV2:
```typescript
// frontend/src/App.tsx
import DashboardV2 from './components/DashboardV2';
// Replace <Dashboard /> with <DashboardV2 />
```
### 4. Run migration and rebuild:
```bash
ssh pi@starship2
cd ~/fidelity
# Stop containers
docker compose down
# Rebuild
docker compose build --no-cache backend frontend
# Start
docker compose up -d
# Run migration
docker compose exec backend alembic upgrade head
# Verify table was created
docker compose exec postgres psql -U fidelity -d fidelitytracker -c "\d market_prices"
```
## Testing
### Test the improved dashboard:
```bash
# 1. Open dashboard - should load instantly with cached data
open http://starship2:3000
# 2. Check logs - should see cache HITs, not Yahoo Finance requests
docker compose logs backend | grep -i "cache\|yahoo"
# 3. Click "Refresh Prices" button
# Should see rate-limited requests in logs
# 4. Check database cache
docker compose exec postgres psql -U fidelity -d fidelitytracker -c "SELECT symbol, price, fetched_at FROM market_prices ORDER BY fetched_at DESC LIMIT 10;"
```
### Test API endpoints directly:
```bash
# Fast load with cache only
curl "http://localhost:8000/api/analytics/overview/1?refresh_prices=false&max_api_calls=0"
# Fresh data with limited API calls
curl "http://localhost:8000/api/analytics/overview/1?refresh_prices=true&max_api_calls=5"
# Manual refresh
curl -X POST "http://localhost:8000/api/analytics/refresh-prices/1?max_api_calls=10"
# Background refresh (returns immediately)
curl -X POST "http://localhost:8000/api/analytics/refresh-prices-background/1?max_api_calls=15"
```
## Benefits
### Before:
- ❌ Dashboard blocked for 30+ seconds
- ❌ Hit rate limits constantly (429 errors)
- ❌ Lost all cache data on restart
- ❌ No way to control API usage
- ❌ Poor user experience
### After:
- ✅ Dashboard loads instantly (<1 second)
- ✅ Respects rate limits with exponential backoff
- ✅ Persistent cache across restarts
- ✅ Configurable API call limits
- ✅ Shows stale data while refreshing
- ✅ Manual refresh option
- ✅ Background updates
- ✅ Cache statistics visible to user
## Maintenance
### Periodic cache refresh (optional):
```bash
# Add to crontab for periodic refresh
*/10 * * * * curl -X POST "http://localhost:8000/api/analytics/refresh-stale-cache?min_age_minutes=10&limit=20"
```
### Clear old cache:
```bash
# Monthly cleanup
curl -X DELETE "http://localhost:8000/api/analytics/clear-old-cache?older_than_days=30"
```
## Future Enhancements
1. **WebSocket updates**: Push price updates to frontend in real-time
2. **Batch updates**: Update all accounts' prices in background job
3. **Multiple data sources**: Fall back to alternative APIs if Yahoo fails
4. **Historical caching**: Store price history for charting
5. **Smart refresh**: Only refresh prices during market hours
## Troubleshooting
### Still getting 429 errors:
- Increase `_rate_limit_delay` in `MarketDataService`
- Decrease `max_api_calls` in API requests
- Use longer `cache_ttl` (e.g., 600 seconds = 10 minutes)
### Dashboard shows old data:
- Check `cache_ttl` setting
- Click "Refresh Prices" button
- Check database: `SELECT * FROM market_prices;`
### Prices not updating:
- Check backend logs for errors
- Verify migration ran: `\d market_prices` in postgres
- Check if symbols are valid (Yahoo Finance format)
## Summary
This solution provides a production-ready approach to handling rate-limited APIs with:
- Fast, responsive UI
- Persistent caching
- Graceful degradation
- User control
- Clear feedback
Users get instant dashboard loads with cached data, and can optionally refresh for the latest prices without blocking the UI.