JoeBot/HOALedgerIQ_Website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: Improved duplicate prevention in cast iron scanner

Browse Source 
- Better link normalization and checking
- Skip items already in seen_links with logging
- Clean up state file to last 500 items
- Always mark items as seen (deal or not)
- Added logging for skipped duplicates

Also: eBay scraping temporarily blocked/changed - investigating
This commit is contained in:
olsch01
2026-04-10 16:16:35 -04:00
parent 30703bfd45
commit 4bd829ca8c
9 changed files with 573 additions and 1002 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
agents/sales-lead/integration.log
View File

@@ -2886,3 +2886,19 @@ No new leads found
[Fri Apr 10 11:00:02 EDT 2026] Response size: 7791 bytes
[Fri Apr 10 11:18:09 EDT 2026] ✓ hoaledgeriq.com/api/calc-submissions responding
[Fri Apr 10 11:18:09 EDT 2026] Response size: 7791 bytes
[Fri Apr 10 12:00:02 EDT 2026] ✓ hoaledgeriq.com/api/calc-submissions responding
[Fri Apr 10 12:00:02 EDT 2026] Response size: 7791 bytes
[Fri Apr 10 12:19:24 EDT 2026] ✓ hoaledgeriq.com/api/calc-submissions responding
[Fri Apr 10 12:19:24 EDT 2026] Response size: 7791 bytes
[Fri Apr 10 13:00:02 EDT 2026] ✓ hoaledgeriq.com/api/calc-submissions responding
[Fri Apr 10 13:00:02 EDT 2026] Response size: 7791 bytes
[Fri Apr 10 14:00:02 EDT 2026] ✓ hoaledgeriq.com/api/calc-submissions responding
[Fri Apr 10 14:00:02 EDT 2026] Response size: 7791 bytes
[Fri Apr 10 14:15:36 EDT 2026] ✓ hoaledgeriq.com/api/calc-submissions responding
[Fri Apr 10 14:15:36 EDT 2026] Response size: 7791 bytes
[Fri Apr 10 15:00:02 EDT 2026] ✓ hoaledgeriq.com/api/calc-submissions responding
[Fri Apr 10 15:00:02 EDT 2026] Response size: 7791 bytes
[Fri Apr 10 15:24:12 EDT 2026] ✓ hoaledgeriq.com/api/calc-submissions responding
[Fri Apr 10 15:24:12 EDT 2026] Response size: 7791 bytes
[Fri Apr 10 16:00:01 EDT 2026] ✓ hoaledgeriq.com/api/calc-submissions responding
[Fri Apr 10 16:00:01 EDT 2026] Response size: 7791 bytes