fix: Improved duplicate prevention in cast iron scanner
- Better link normalization and checking - Skip items already in seen_links with logging - Clean up state file to last 500 items - Always mark items as seen (deal or not) - Added logging for skipped duplicates Also: eBay scraping temporarily blocked/changed - investigating
This commit is contained in:
@@ -44,6 +44,10 @@
|
||||
"deep-scraper": {
|
||||
"version": "1.0.1",
|
||||
"installedAt": 1775821381724
|
||||
},
|
||||
"marketplace": {
|
||||
"version": "1.0.1",
|
||||
"installedAt": 1775850557338
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user