feat: Cast Iron Scout multi-channel expansion
- Added Craigslist scanner framework - Added Facebook Marketplace placeholder - Updated main scanner to aggregate all sources - Added STATUS.md for development tracking - Fixed import paths for all scanners - Ready for HTML scraping implementation Current status: - eBay: RSS built but unreliable, need HTML scraping - Craigslist: Framework ready, debugging HTML parsing - Facebook: Placeholder (needs Selenium) - All sources tracked in unified scan loop
This commit is contained in:
77
agents/cast-iron-scout/STATUS.md
Normal file
77
agents/cast-iron-scout/STATUS.md
Normal file
@@ -0,0 +1,77 @@
|
|||||||
|
# Cast Iron Scout - Development Status
|
||||||
|
|
||||||
|
**Last Updated:** April 9, 2026 - 5:50 PM
|
||||||
|
**Developer:** Forge
|
||||||
|
|
||||||
|
## ✅ Completed
|
||||||
|
|
||||||
|
- [x] Core scanner engine built
|
||||||
|
- [x] Valuation engine with brand/size/type logic
|
||||||
|
- [x] Telegram alert system
|
||||||
|
- [x] Duplicate prevention (state tracking)
|
||||||
|
- [x] Hourly cron job configured
|
||||||
|
- [x] Configuration system
|
||||||
|
- [x] Logging system
|
||||||
|
|
||||||
|
## 🚧 In Progress
|
||||||
|
|
||||||
|
### Channel: eBay
|
||||||
|
- [x] RSS feed scanner (built)
|
||||||
|
- [ ] HTML scraper (needed - RSS unreliable)
|
||||||
|
- **Status:** ⚠️ RSS feeds not returning data consistently
|
||||||
|
- **Next:** Implement proper HTML scraping with BeautifulSoup
|
||||||
|
|
||||||
|
### Channel: Craigslist
|
||||||
|
- [x] Scanner framework built
|
||||||
|
- [ ] HTML parsing (in progress)
|
||||||
|
- [ ] Multi-city scanning
|
||||||
|
- **Status:** ⚠️ Returning 0 items - needs HTML structure fix
|
||||||
|
- **Next:** Debug HTML parsing, add proper result extraction
|
||||||
|
|
||||||
|
### Channel: Facebook Marketplace
|
||||||
|
- [x] Scanner framework created
|
||||||
|
- [ ] Selenium/Playwright implementation
|
||||||
|
- [ ] Location-based searches
|
||||||
|
- **Status:** 📝 Placeholder only (requires headless browser)
|
||||||
|
- **Next:** Install Selenium, implement headless Chrome scraping
|
||||||
|
|
||||||
|
### Channel: EstateSale.com
|
||||||
|
- [ ] Not started
|
||||||
|
- **Status:** ⏳ Backlog
|
||||||
|
- **Next:** Research site structure, build scanner
|
||||||
|
|
||||||
|
## 🔧 Technical Debt
|
||||||
|
|
||||||
|
1. **eBay RSS unreliable** - Need HTML scraping
|
||||||
|
2. **Craigslist HTML parsing** - Need to fix selector logic
|
||||||
|
3. **Facebook requires Selenium** - Heavy dependency, slow scans
|
||||||
|
4. **No image recognition yet** - Can't identify logos from photos
|
||||||
|
5. **No price history database** - Can't track sold items
|
||||||
|
|
||||||
|
## 📊 Performance
|
||||||
|
|
||||||
|
- **Scan frequency:** Every hour
|
||||||
|
- **Sources active:** 0/3 (all in development)
|
||||||
|
- **Deals found:** 0 (no working sources yet)
|
||||||
|
- **False positives:** N/A
|
||||||
|
|
||||||
|
## 🎯 Next Steps (Priority Order)
|
||||||
|
|
||||||
|
1. **Fix eBay scanner** - HTML scraping (2-3 hours)
|
||||||
|
2. **Fix Craigslist scanner** - Debug HTML parsing (1 hour)
|
||||||
|
3. **Add Facebook with Selenium** - Heavy lift (4-6 hours)
|
||||||
|
4. **Test with real data** - Validate against manual searches
|
||||||
|
5. **Add image recognition** - Logo identification (future)
|
||||||
|
6. **Add sold items database** - Better FMV calculations (future)
|
||||||
|
|
||||||
|
## 📝 Notes
|
||||||
|
|
||||||
|
- Craigslist may block frequent requests - need rate limiting
|
||||||
|
- Facebook Marketplace requires login for full access
|
||||||
|
- eBay has official API but requires approval
|
||||||
|
- Consider adding OfferUp, Letgo as lighter alternatives to FB
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Current Focus:** Getting eBay and Craigslist working reliably
|
||||||
|
**ETA for First Deal Alert:** TBD (depends on scraping fixes)
|
||||||
@@ -9,6 +9,8 @@ import sys
|
|||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from sources.ebay_scanner import search_ebay_cast_iron
|
from sources.ebay_scanner import search_ebay_cast_iron
|
||||||
|
from sources.craigslist_scanner import search_craigslist_cast_iron
|
||||||
|
from sources.facebook_scanner import search_facebook_marketplaceCast_iron
|
||||||
from valuation import is_good_deal, calculate_fmv
|
from valuation import is_good_deal, calculate_fmv
|
||||||
|
|
||||||
SCRIPT_DIR = Path(__file__).parent
|
SCRIPT_DIR = Path(__file__).parent
|
||||||
@@ -80,14 +82,36 @@ def scan_all_sources():
|
|||||||
state = load_state()
|
state = load_state()
|
||||||
seen_links = set(state.get('seen_links', []))
|
seen_links = set(state.get('seen_links', []))
|
||||||
|
|
||||||
|
all_items = []
|
||||||
|
|
||||||
# Scan eBay
|
# Scan eBay
|
||||||
items = search_ebay_cast_iron()
|
try:
|
||||||
log(f"Found {len(items)} items on eBay")
|
ebay_items = search_ebay_cast_iron()
|
||||||
|
log(f"Found {len(ebay_items)} items on eBay")
|
||||||
|
all_items.extend(ebay_items)
|
||||||
|
except Exception as e:
|
||||||
|
log(f"eBay scan error: {e}")
|
||||||
|
|
||||||
|
# Scan Craigslist
|
||||||
|
try:
|
||||||
|
cl_items = search_craigslist_cast_iron()
|
||||||
|
log(f"Found {len(cl_items)} items on Craigslist")
|
||||||
|
all_items.extend(cl_items)
|
||||||
|
except Exception as e:
|
||||||
|
log(f"Craigslist scan error: {e}")
|
||||||
|
|
||||||
|
# Scan Facebook Marketplace (placeholder for now)
|
||||||
|
try:
|
||||||
|
fb_items = search_facebook_marketplaceCast_iron(config)
|
||||||
|
log(f"Found {len(fb_items)} items on Facebook Marketplace")
|
||||||
|
all_items.extend(fb_items)
|
||||||
|
except Exception as e:
|
||||||
|
log(f"Facebook scan error: {e}")
|
||||||
|
|
||||||
deals_found = 0
|
deals_found = 0
|
||||||
min_discount = config.get('min_discount_percent', 50)
|
min_discount = config.get('min_discount_percent', 50)
|
||||||
|
|
||||||
for item in items:
|
for item in all_items:
|
||||||
# Skip if already seen
|
# Skip if already seen
|
||||||
if item['link'] in seen_links:
|
if item['link'] in seen_links:
|
||||||
continue
|
continue
|
||||||
@@ -110,7 +134,7 @@ def scan_all_sources():
|
|||||||
state['seen_links'] = list(seen_links)
|
state['seen_links'] = list(seen_links)
|
||||||
save_state(state)
|
save_state(state)
|
||||||
|
|
||||||
log(f"Scan complete. Deals found: {deals_found}, Total items processed: {len(items)}")
|
log(f"Scan complete. Deals found: {deals_found}, Total items processed: {len(all_items)}")
|
||||||
return deals_found
|
return deals_found
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|||||||
87
agents/cast-iron-scout/sources/craigslist_scanner.py
Normal file
87
agents/cast-iron-scout/sources/craigslist_scanner.py
Normal file
@@ -0,0 +1,87 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Craigslist Scanner for Cast Iron
|
||||||
|
Scans Craigslist for cast iron cookware deals
|
||||||
|
"""
|
||||||
|
import requests
|
||||||
|
from datetime import datetime
|
||||||
|
import re
|
||||||
|
|
||||||
|
def search_craigslist_cast_iron(locations=None):
|
||||||
|
"""
|
||||||
|
Search Craigslist for cast iron items
|
||||||
|
locations: list of Craigslist location codes (e.g., 'atl', 'nyc', 'la')
|
||||||
|
"""
|
||||||
|
if locations is None:
|
||||||
|
# Major metro areas with active cast iron markets
|
||||||
|
locations = [
|
||||||
|
'atlanta', 'austin', 'boston', 'charleston', 'chicago',
|
||||||
|
'dallas', 'denver', 'detroit', 'houston', 'kansas',
|
||||||
|
'lasvegas', 'losangeles', 'miami', 'minneapolis', 'nashville',
|
||||||
|
'newjersey', 'newyork', 'orangecounty', 'philadelphia',
|
||||||
|
'phoenix', 'pittsburgh', 'portland', 'raleigh', 'sacramento',
|
||||||
|
'sandiego', 'sf', 'seattle', 'stlouis', 'tampa', 'washingtondc'
|
||||||
|
]
|
||||||
|
|
||||||
|
items = []
|
||||||
|
|
||||||
|
search_query = "cast iron skillet"
|
||||||
|
|
||||||
|
for location in locations[:5]: # Start with first 5 to avoid rate limiting
|
||||||
|
try:
|
||||||
|
url = f"https://{location}.craigslist.org/search/sss?query={search_query.replace(' ', '%20')}"
|
||||||
|
|
||||||
|
response = requests.get(url, headers={
|
||||||
|
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36'
|
||||||
|
}, timeout=10)
|
||||||
|
|
||||||
|
if response.status_code == 200:
|
||||||
|
# Parse HTML for listings
|
||||||
|
# Craigslist structure: each result is in a div.result-row
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
soup = BeautifulSoup(response.text, 'html.parser')
|
||||||
|
|
||||||
|
results = soup.find_all('li', class_='result-row')
|
||||||
|
|
||||||
|
for result in results[:10]: # Top 10 per location
|
||||||
|
try:
|
||||||
|
title_elem = result.find('a', class_='result-title')
|
||||||
|
if not title_elem:
|
||||||
|
continue
|
||||||
|
|
||||||
|
title = title_elem.text
|
||||||
|
link = title_elem['href']
|
||||||
|
price_text = result.find('span', class_='result-price')
|
||||||
|
price = 0
|
||||||
|
|
||||||
|
if price_text:
|
||||||
|
price_match = re.search(r'\$?([\d,]+)', price_text.text)
|
||||||
|
if price_match:
|
||||||
|
price = float(price_match.group(1).replace(',', ''))
|
||||||
|
|
||||||
|
# Extract location
|
||||||
|
loc_elem = result.find('span', class_='result-hood')
|
||||||
|
loc = loc_elem.text.strip() if loc_elem else location
|
||||||
|
|
||||||
|
items.append({
|
||||||
|
'title': title,
|
||||||
|
'price': price,
|
||||||
|
'link': link,
|
||||||
|
'source': f'Craigslist ({location})',
|
||||||
|
'location': loc,
|
||||||
|
'found_at': datetime.now().isoformat()
|
||||||
|
})
|
||||||
|
except Exception as e:
|
||||||
|
continue
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Error scanning Craigslist {location}: {e}")
|
||||||
|
|
||||||
|
return items
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
print("🔍 Scanning Craigslist for cast iron...")
|
||||||
|
items = search_craigslist_cast_iron()
|
||||||
|
print(f"Found {len(items)} items")
|
||||||
|
for item in items[:5]:
|
||||||
|
print(f" - {item['title'][:50]} - ${item['price']} ({item['source']})")
|
||||||
40
agents/cast-iron-scout/sources/facebook_scanner.py
Normal file
40
agents/cast-iron-scout/sources/facebook_scanner.py
Normal file
@@ -0,0 +1,40 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Facebook Marketplace Scanner
|
||||||
|
Scans FB Marketplace for local cast iron deals
|
||||||
|
Note: Requires Selenium for now (FB has no public API)
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
# Facebook Marketplace doesn't have RSS or public API
|
||||||
|
# This is a placeholder for when we implement Selenium/Playwright
|
||||||
|
# For now, we'll use manual URL monitoring
|
||||||
|
|
||||||
|
def search_facebook_marketplaceCast_iron(config, location_radius=50):
|
||||||
|
"""
|
||||||
|
Search Facebook Marketplace for cast iron
|
||||||
|
This will eventually use Selenium to scrape FB Marketplace
|
||||||
|
|
||||||
|
For now, returns empty list - will be implemented with:
|
||||||
|
- Selenium WebDriver (headless Chrome)
|
||||||
|
- Location-based searches
|
||||||
|
- Image extraction
|
||||||
|
"""
|
||||||
|
items = []
|
||||||
|
|
||||||
|
# TODO: Implement Selenium scraper
|
||||||
|
# Search URLs to monitor:
|
||||||
|
# https://www.facebook.com/marketplace/search?query=cast%20iron%20skillet
|
||||||
|
# https://www.facebook.com/marketplace/search?query=griswold
|
||||||
|
# https://www.facebook.com/marketplace/search?query=wagner%20cast%20iron
|
||||||
|
|
||||||
|
print("📘 Facebook Marketplace scanner: Pending Selenium implementation")
|
||||||
|
print(" Manual check: https://www.facebook.com/marketplace/search?query=cast%20iron")
|
||||||
|
|
||||||
|
return items
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
print("Testing Facebook Marketplace scanner...")
|
||||||
|
items = search_facebook_marketplaceCast_iron({})
|
||||||
|
print(f"Found {len(items)} items")
|
||||||
Reference in New Issue
Block a user