feat: Cast Iron Scout multi-channel expansion
- Added Craigslist scanner framework - Added Facebook Marketplace placeholder - Updated main scanner to aggregate all sources - Added STATUS.md for development tracking - Fixed import paths for all scanners - Ready for HTML scraping implementation Current status: - eBay: RSS built but unreliable, need HTML scraping - Craigslist: Framework ready, debugging HTML parsing - Facebook: Placeholder (needs Selenium) - All sources tracked in unified scan loop
This commit is contained in:
77
agents/cast-iron-scout/STATUS.md
Normal file
77
agents/cast-iron-scout/STATUS.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# Cast Iron Scout - Development Status
|
||||
|
||||
**Last Updated:** April 9, 2026 - 5:50 PM
|
||||
**Developer:** Forge
|
||||
|
||||
## ✅ Completed
|
||||
|
||||
- [x] Core scanner engine built
|
||||
- [x] Valuation engine with brand/size/type logic
|
||||
- [x] Telegram alert system
|
||||
- [x] Duplicate prevention (state tracking)
|
||||
- [x] Hourly cron job configured
|
||||
- [x] Configuration system
|
||||
- [x] Logging system
|
||||
|
||||
## 🚧 In Progress
|
||||
|
||||
### Channel: eBay
|
||||
- [x] RSS feed scanner (built)
|
||||
- [ ] HTML scraper (needed - RSS unreliable)
|
||||
- **Status:** ⚠️ RSS feeds not returning data consistently
|
||||
- **Next:** Implement proper HTML scraping with BeautifulSoup
|
||||
|
||||
### Channel: Craigslist
|
||||
- [x] Scanner framework built
|
||||
- [ ] HTML parsing (in progress)
|
||||
- [ ] Multi-city scanning
|
||||
- **Status:** ⚠️ Returning 0 items - needs HTML structure fix
|
||||
- **Next:** Debug HTML parsing, add proper result extraction
|
||||
|
||||
### Channel: Facebook Marketplace
|
||||
- [x] Scanner framework created
|
||||
- [ ] Selenium/Playwright implementation
|
||||
- [ ] Location-based searches
|
||||
- **Status:** 📝 Placeholder only (requires headless browser)
|
||||
- **Next:** Install Selenium, implement headless Chrome scraping
|
||||
|
||||
### Channel: EstateSale.com
|
||||
- [ ] Not started
|
||||
- **Status:** ⏳ Backlog
|
||||
- **Next:** Research site structure, build scanner
|
||||
|
||||
## 🔧 Technical Debt
|
||||
|
||||
1. **eBay RSS unreliable** - Need HTML scraping
|
||||
2. **Craigslist HTML parsing** - Need to fix selector logic
|
||||
3. **Facebook requires Selenium** - Heavy dependency, slow scans
|
||||
4. **No image recognition yet** - Can't identify logos from photos
|
||||
5. **No price history database** - Can't track sold items
|
||||
|
||||
## 📊 Performance
|
||||
|
||||
- **Scan frequency:** Every hour
|
||||
- **Sources active:** 0/3 (all in development)
|
||||
- **Deals found:** 0 (no working sources yet)
|
||||
- **False positives:** N/A
|
||||
|
||||
## 🎯 Next Steps (Priority Order)
|
||||
|
||||
1. **Fix eBay scanner** - HTML scraping (2-3 hours)
|
||||
2. **Fix Craigslist scanner** - Debug HTML parsing (1 hour)
|
||||
3. **Add Facebook with Selenium** - Heavy lift (4-6 hours)
|
||||
4. **Test with real data** - Validate against manual searches
|
||||
5. **Add image recognition** - Logo identification (future)
|
||||
6. **Add sold items database** - Better FMV calculations (future)
|
||||
|
||||
## 📝 Notes
|
||||
|
||||
- Craigslist may block frequent requests - need rate limiting
|
||||
- Facebook Marketplace requires login for full access
|
||||
- eBay has official API but requires approval
|
||||
- Consider adding OfferUp, Letgo as lighter alternatives to FB
|
||||
|
||||
---
|
||||
|
||||
**Current Focus:** Getting eBay and Craigslist working reliably
|
||||
**ETA for First Deal Alert:** TBD (depends on scraping fixes)
|
||||
@@ -9,6 +9,8 @@ import sys
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from sources.ebay_scanner import search_ebay_cast_iron
|
||||
from sources.craigslist_scanner import search_craigslist_cast_iron
|
||||
from sources.facebook_scanner import search_facebook_marketplaceCast_iron
|
||||
from valuation import is_good_deal, calculate_fmv
|
||||
|
||||
SCRIPT_DIR = Path(__file__).parent
|
||||
@@ -80,14 +82,36 @@ def scan_all_sources():
|
||||
state = load_state()
|
||||
seen_links = set(state.get('seen_links', []))
|
||||
|
||||
all_items = []
|
||||
|
||||
# Scan eBay
|
||||
items = search_ebay_cast_iron()
|
||||
log(f"Found {len(items)} items on eBay")
|
||||
try:
|
||||
ebay_items = search_ebay_cast_iron()
|
||||
log(f"Found {len(ebay_items)} items on eBay")
|
||||
all_items.extend(ebay_items)
|
||||
except Exception as e:
|
||||
log(f"eBay scan error: {e}")
|
||||
|
||||
# Scan Craigslist
|
||||
try:
|
||||
cl_items = search_craigslist_cast_iron()
|
||||
log(f"Found {len(cl_items)} items on Craigslist")
|
||||
all_items.extend(cl_items)
|
||||
except Exception as e:
|
||||
log(f"Craigslist scan error: {e}")
|
||||
|
||||
# Scan Facebook Marketplace (placeholder for now)
|
||||
try:
|
||||
fb_items = search_facebook_marketplaceCast_iron(config)
|
||||
log(f"Found {len(fb_items)} items on Facebook Marketplace")
|
||||
all_items.extend(fb_items)
|
||||
except Exception as e:
|
||||
log(f"Facebook scan error: {e}")
|
||||
|
||||
deals_found = 0
|
||||
min_discount = config.get('min_discount_percent', 50)
|
||||
|
||||
for item in items:
|
||||
for item in all_items:
|
||||
# Skip if already seen
|
||||
if item['link'] in seen_links:
|
||||
continue
|
||||
@@ -110,7 +134,7 @@ def scan_all_sources():
|
||||
state['seen_links'] = list(seen_links)
|
||||
save_state(state)
|
||||
|
||||
log(f"Scan complete. Deals found: {deals_found}, Total items processed: {len(items)}")
|
||||
log(f"Scan complete. Deals found: {deals_found}, Total items processed: {len(all_items)}")
|
||||
return deals_found
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
87
agents/cast-iron-scout/sources/craigslist_scanner.py
Normal file
87
agents/cast-iron-scout/sources/craigslist_scanner.py
Normal file
@@ -0,0 +1,87 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Craigslist Scanner for Cast Iron
|
||||
Scans Craigslist for cast iron cookware deals
|
||||
"""
|
||||
import requests
|
||||
from datetime import datetime
|
||||
import re
|
||||
|
||||
def search_craigslist_cast_iron(locations=None):
|
||||
"""
|
||||
Search Craigslist for cast iron items
|
||||
locations: list of Craigslist location codes (e.g., 'atl', 'nyc', 'la')
|
||||
"""
|
||||
if locations is None:
|
||||
# Major metro areas with active cast iron markets
|
||||
locations = [
|
||||
'atlanta', 'austin', 'boston', 'charleston', 'chicago',
|
||||
'dallas', 'denver', 'detroit', 'houston', 'kansas',
|
||||
'lasvegas', 'losangeles', 'miami', 'minneapolis', 'nashville',
|
||||
'newjersey', 'newyork', 'orangecounty', 'philadelphia',
|
||||
'phoenix', 'pittsburgh', 'portland', 'raleigh', 'sacramento',
|
||||
'sandiego', 'sf', 'seattle', 'stlouis', 'tampa', 'washingtondc'
|
||||
]
|
||||
|
||||
items = []
|
||||
|
||||
search_query = "cast iron skillet"
|
||||
|
||||
for location in locations[:5]: # Start with first 5 to avoid rate limiting
|
||||
try:
|
||||
url = f"https://{location}.craigslist.org/search/sss?query={search_query.replace(' ', '%20')}"
|
||||
|
||||
response = requests.get(url, headers={
|
||||
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36'
|
||||
}, timeout=10)
|
||||
|
||||
if response.status_code == 200:
|
||||
# Parse HTML for listings
|
||||
# Craigslist structure: each result is in a div.result-row
|
||||
from bs4 import BeautifulSoup
|
||||
soup = BeautifulSoup(response.text, 'html.parser')
|
||||
|
||||
results = soup.find_all('li', class_='result-row')
|
||||
|
||||
for result in results[:10]: # Top 10 per location
|
||||
try:
|
||||
title_elem = result.find('a', class_='result-title')
|
||||
if not title_elem:
|
||||
continue
|
||||
|
||||
title = title_elem.text
|
||||
link = title_elem['href']
|
||||
price_text = result.find('span', class_='result-price')
|
||||
price = 0
|
||||
|
||||
if price_text:
|
||||
price_match = re.search(r'\$?([\d,]+)', price_text.text)
|
||||
if price_match:
|
||||
price = float(price_match.group(1).replace(',', ''))
|
||||
|
||||
# Extract location
|
||||
loc_elem = result.find('span', class_='result-hood')
|
||||
loc = loc_elem.text.strip() if loc_elem else location
|
||||
|
||||
items.append({
|
||||
'title': title,
|
||||
'price': price,
|
||||
'link': link,
|
||||
'source': f'Craigslist ({location})',
|
||||
'location': loc,
|
||||
'found_at': datetime.now().isoformat()
|
||||
})
|
||||
except Exception as e:
|
||||
continue
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error scanning Craigslist {location}: {e}")
|
||||
|
||||
return items
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("🔍 Scanning Craigslist for cast iron...")
|
||||
items = search_craigslist_cast_iron()
|
||||
print(f"Found {len(items)} items")
|
||||
for item in items[:5]:
|
||||
print(f" - {item['title'][:50]} - ${item['price']} ({item['source']})")
|
||||
40
agents/cast-iron-scout/sources/facebook_scanner.py
Normal file
40
agents/cast-iron-scout/sources/facebook_scanner.py
Normal file
@@ -0,0 +1,40 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Facebook Marketplace Scanner
|
||||
Scans FB Marketplace for local cast iron deals
|
||||
Note: Requires Selenium for now (FB has no public API)
|
||||
"""
|
||||
import re
|
||||
from datetime import datetime
|
||||
|
||||
# Facebook Marketplace doesn't have RSS or public API
|
||||
# This is a placeholder for when we implement Selenium/Playwright
|
||||
# For now, we'll use manual URL monitoring
|
||||
|
||||
def search_facebook_marketplaceCast_iron(config, location_radius=50):
|
||||
"""
|
||||
Search Facebook Marketplace for cast iron
|
||||
This will eventually use Selenium to scrape FB Marketplace
|
||||
|
||||
For now, returns empty list - will be implemented with:
|
||||
- Selenium WebDriver (headless Chrome)
|
||||
- Location-based searches
|
||||
- Image extraction
|
||||
"""
|
||||
items = []
|
||||
|
||||
# TODO: Implement Selenium scraper
|
||||
# Search URLs to monitor:
|
||||
# https://www.facebook.com/marketplace/search?query=cast%20iron%20skillet
|
||||
# https://www.facebook.com/marketplace/search?query=griswold
|
||||
# https://www.facebook.com/marketplace/search?query=wagner%20cast%20iron
|
||||
|
||||
print("📘 Facebook Marketplace scanner: Pending Selenium implementation")
|
||||
print(" Manual check: https://www.facebook.com/marketplace/search?query=cast%20iron")
|
||||
|
||||
return items
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("Testing Facebook Marketplace scanner...")
|
||||
items = search_facebook_marketplaceCast_iron({})
|
||||
print(f"Found {len(items)} items")
|
||||
Reference in New Issue
Block a user