fix: Improved duplicate prevention in cast iron scanner
- Better link normalization and checking - Skip items already in seen_links with logging - Clean up state file to last 500 items - Always mark items as seen (deal or not) - Added logging for skipped duplicates Also: eBay scraping temporarily blocked/changed - investigating
This commit is contained in:
@@ -114,8 +114,15 @@
|
||||
"1sgqzb9",
|
||||
"1sgosjw",
|
||||
"1shf8nt",
|
||||
"1sh34bf"
|
||||
"1sh34bf",
|
||||
"1shvh2y",
|
||||
"1shtzie",
|
||||
"1shsg8u",
|
||||
"1shr186",
|
||||
"1shqvf5",
|
||||
"1shn1jg",
|
||||
"1shm8fy"
|
||||
],
|
||||
"total_scanned": 1400,
|
||||
"total_matches": 35
|
||||
"total_scanned": 1450,
|
||||
"total_matches": 39
|
||||
}
|
||||
Reference in New Issue
Block a user