fix: Improved duplicate prevention in cast iron scanner

- Better link normalization and checking
- Skip items already in seen_links with logging
- Clean up state file to last 500 items
- Always mark items as seen (deal or not)
- Added logging for skipped duplicates

Also: eBay scraping temporarily blocked/changed - investigating
This commit is contained in:
2026-04-10 16:16:35 -04:00
parent 30703bfd45
commit 4bd829ca8c
9 changed files with 573 additions and 1002 deletions

View File

@@ -1,7 +1,7 @@
{
"processed_leads": [],
"processed_calc_ids": [1, 2, 3, 4],
"last_check": "2026-04-10T15:18:09Z",
"last_check": "2026-04-10T20:00:01Z",
"status": "active",
"notes": "Hourly monitoring enabled. Next check in 60 minutes."
}