fix: Improved duplicate prevention in cast iron scanner

- Better link normalization and checking
- Skip items already in seen_links with logging
- Clean up state file to last 500 items
- Always mark items as seen (deal or not)
- Added logging for skipped duplicates

Also: eBay scraping temporarily blocked/changed - investigating
This commit is contained in:
2026-04-10 16:16:35 -04:00
parent 30703bfd45
commit 4bd829ca8c
9 changed files with 573 additions and 1002 deletions

View File

@@ -1,5 +1,5 @@
{
"last_check": "2026-04-10T10:09:06.314350",
"last_check": "2026-04-10T13:10:34.858296",
"processed": 0,
"upgraded": 0,
"processed_ids": [