Arg Parsing Refactor + Support for YAML Config Support (take 2!) (#59)

* Create an argument parser class

* move constants, arg parser to separate files in utils/*

* ensure yaml config overriden by command-line args

* yaml loading work:
- simplify yaml config by using yargs.config option
- move all option parsing to argParser, simply expose parseArgs
- export constants directly
- add lint to util/* files

* support inline 'seeds' in cmdline and yaml config

tests:
- add test for crawl config, ensuring seeds crawled + wacz created
- add test to ensure cmdline overrides yaml config

* scope fix: empty scope implies only fixed list, use '.*' for any scope

* lint fix

* update readme with yaml config info

* allow 'url' and 'seeds' if both provided

Co-authored-by: Emma Dickson <emmadickson@Emmas-MacBook-Pro.local>
Co-authored-by: emmadickson <emma.dickson@artsymail.com>
This commit is contained in:
Ilya Kreymer
2021-06-23 19:36:32 -07:00
committed by Ilya Kreymer
parent ae4ce979fb
commit 3ebe511b32
13 changed files with 570 additions and 393 deletions

View File

@@ -1,4 +1,4 @@
#pywb>=2.5.0
git+https://github.com/webrecorder/pywb@main
pywb>=2.6.0b2
#git+https://github.com/webrecorder/pywb@main
uwsgi
wacz>=0.3.0