fix: Improved duplicate prevention in cast iron scanner

- Better link normalization and checking
- Skip items already in seen_links with logging
- Clean up state file to last 500 items
- Always mark items as seen (deal or not)
- Added logging for skipped duplicates

Also: eBay scraping temporarily blocked/changed - investigating
This commit is contained in:
2026-04-10 16:16:35 -04:00
parent 30703bfd45
commit 4bd829ca8c
9 changed files with 573 additions and 1002 deletions

View File

@@ -114,8 +114,15 @@
"1sgqzb9",
"1sgosjw",
"1shf8nt",
"1sh34bf"
"1sh34bf",
"1shvh2y",
"1shtzie",
"1shsg8u",
"1shr186",
"1shqvf5",
"1shn1jg",
"1shm8fy"
],
"total_scanned": 1400,
"total_matches": 35
"total_scanned": 1450,
"total_matches": 39
}