Mirrors/browsertrix-crawler
1
0
Fork 0
mirror of https://github.com/webrecorder/browsertrix-crawler.git synced 2025-12-25 11:20:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

Support for per-URL conditional Block Rules (#68)

Browse Source 
- Support for block rules specified in YAML config to exclude URLs based on regex, and also negate a rule by specifying `allowOnly` to allow URLs based on certain regex.
- Support for conditional blocking for iframes, based on content of iframe text, specified via frameTextMatch regex.
- Support for restricting block rules based on containing frame URL, specified via inFrameURL param.
- Testing for various blockRules configurations
- Fixes Support URL-level WARC-writing inclusion/exclusion lists #15
- optional message to add when a URL is blocked, specified via 'blockMessage'
- update README for blockRules
- bump to pywb dependency 2.5.0b4
This commit is contained in:
Ilya Kreymer
2021-07-19 15:49:43 -07:00
committed by Ilya Kreymer
parent 838e1fa1bd
commit 6dbdff9656
8 changed files with 363 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
requirements.txt
View File

@@ -1,4 +1,4 @@
pywb>=2.6.0b3
#git+https://github.com/webrecorder/pywb@main
pywb>=2.6.0b4
git+https://github.com/webrecorder/pywb@main
uwsgi
wacz>=0.3.0