mirror of
https://github.com/webrecorder/browsertrix-crawler.git
synced 2025-12-26 03:40:19 +00:00
Arg Parsing Refactor + Support for YAML Config Support (take 2!) (#59)
* Create an argument parser class * move constants, arg parser to separate files in utils/* * ensure yaml config overriden by command-line args * yaml loading work: - simplify yaml config by using yargs.config option - move all option parsing to argParser, simply expose parseArgs - export constants directly - add lint to util/* files * support inline 'seeds' in cmdline and yaml config tests: - add test for crawl config, ensuring seeds crawled + wacz created - add test to ensure cmdline overrides yaml config * scope fix: empty scope implies only fixed list, use '.*' for any scope * lint fix * update readme with yaml config info * allow 'url' and 'seeds' if both provided Co-authored-by: Emma Dickson <emmadickson@Emmas-MacBook-Pro.local> Co-authored-by: emmadickson <emma.dickson@artsymail.com>
This commit is contained in:
committed by
Ilya Kreymer
parent
ae4ce979fb
commit
3ebe511b32
@@ -1,4 +1,4 @@
|
||||
#pywb>=2.5.0
|
||||
git+https://github.com/webrecorder/pywb@main
|
||||
pywb>=2.6.0b2
|
||||
#git+https://github.com/webrecorder/pywb@main
|
||||
uwsgi
|
||||
wacz>=0.3.0
|
||||
|
||||
Reference in New Issue
Block a user