fix: Improved duplicate prevention in cast iron scanner

- Better link normalization and checking
- Skip items already in seen_links with logging
- Clean up state file to last 500 items
- Always mark items as seen (deal or not)
- Added logging for skipped duplicates

Also: eBay scraping temporarily blocked/changed - investigating
This commit is contained in:
2026-04-10 16:16:35 -04:00
parent 30703bfd45
commit 4bd829ca8c
9 changed files with 573 additions and 1002 deletions

View File

@@ -44,6 +44,10 @@
"deep-scraper": {
"version": "1.0.1",
"installedAt": 1775821381724
},
"marketplace": {
"version": "1.0.1",
"installedAt": 1775850557338
}
}
}