Files
browsertrix-crawler/tests/dedup-basic.test.js
Ilya Kreymer e0244391f1 update to new data model:
- hashes stored in separate crawl specific entries, h:<crawlid>
- wacz files stored in crawl specific list, c:<crawlid>:wacz
- hashes committed to 'alldupes' hashset when crawl is complete, crawls added to 'allcrawls' set
- store filename, crawlId in related.requires list entries for each wacz
2025-12-11 10:43:57 -08:00

4.6 KiB