The "npm:validate" task validates the repository's `package.json` npm manifest file against its JSON schema to catch any
problems with its data format.
In order to avoid duplication of content, JSON schemas may reference other schemas via the `$ref` keyword. The
`package.json` schema was recently updated to share resources with the "base" configuration schema, which caused the
validation to start failing:
schema /tmp/package-json-schema-m3x8Uu4NED.json is invalid
error: can't resolve reference https://json.schemastore.org/base.json#/definitions/license from id https://json.schemastore.org/package.json#
The solution is to configure the task to download that schema as well and also to provide its path to the avj-cli
validator via a `-r` flag.
The "npm:validate" task validates the repository's `package.json` npm manifest file against its JSON schema to catch any
problems with its data format.
In order to avoid duplication of content, JSON schemas may reference other schemas via the `$ref` keyword. The
`package.json` schema was recently updated to share resources with the npm-badges configuration schema, which caused the
validation to start failing:
schema /tmp/package-json-schema-WItZVgRyuR.json is invalid
error: can't resolve reference https://json.schemastore.org/partial-eslint-plugins.json from id https://json.schemastore.org/eslintrc.json#
task: Failed to run task "npm:validate": exit status 1
The solution is to configure the workflow to download that schema as well and also to provide its path to the avj-cli
validator via a `-r` flag.
Previously a task named ts:install-deps was used to install the npm-managed dependencies of the project. At the time the
task was added, npm was used to manage the action's TypeScript code dependencies as well as TypeScript-specific
development tool dependencies. For this reason, it seemed reasonable to use the "ts" prefix on the task name to place it
under the TypeScript "namespace" of tasks.
In addition to the TypeScript-specific dependencies, npm is now used to manage tool dependencies for general management
of the project not specific to TypeScript. For this reason, and because it is now the standardized task name used by the
reusable "template" assets, it is better to use the task with the more appropriate "npm" prefix.
Some of the assets use tools sourced from the npm software registry.
Previously, the version of the tools used was not controlled. This was problematic because:
- A different version of the tool may be used on the contributor's machine than on the CI runner, resulting in confusing
failures.
- The project is immediately subject to disruption or breakage resulting from a release of the tool.
---
These tools were installed via either of the following methods:
`npx <pkg>`
This approach has the following behaviors of interest:
https://docs.npmjs.com/cli/v8/commands/npx#description
> If any requested packages are not present in the local project dependencies, then they are installed to a folder in the npm cache, which is added to the PATH environment variable in the executed process.
> Package names provided without a specifier will be matched with whatever version exists in the local project. Package names with a specifier will only be considered a match if they have the exact same name and version as the local dependency.
This means that the version used was:
1. Whatever happens to be present in the local cache
2. The latest available version if it is not already present
`npm install --global <pkg>`
The latest available version of the package is used.
---
The new approach is to specify the version of the tools via the standard npm metadata files (package.json +
package-lock.json). This approach was chosen over the `npx <pkg>@<version>` alternative for the following reasons:
- Enables automated updates via Dependabot PRs
- Enables automated vulnerability alerts
- Separates dependency management from the asset contents (i.e., no need to mess with the taskfile or workflow on every
update)
- Matches how we are already managing Python dependencies (pyproject.toml + poetry.lock)
The "Check npm" GitHub Actions workflow validates the repository's `package.json` npm manifest file against its JSON
schema to catch any problems with its data format.
In order to avoid duplication of content, JSON schemas may reference other schemas via the `$ref` keyword. The
`package.json` schema was recently updated to share resources with the npm-badges configuration schema, which caused the
validation to start failing:
schema /home/runner/work/_temp/json-schema/package-json-schema.json is invalid
error: can't resolve reference https://json.schemastore.org/npm-badges.json from id #
The solution is to configure the workflow to download that schema as well and also to provide its path to the avj-cli
validator via an `-r` flag.
In addition to helping others use Task in their GitHub Actions workflows, this project uses Task for its own development.
Most of the repository's GitHub Actions workflows use the approach of executing a task. This approach allows maintaining
a single set of commands that can easily be used locally by contributors in addition to their use by the CI system.
The "Check npm" workflow was an exception to this pattern, using commands directly coded into the workflow file. This
meant the contributor would need to study the workflow and run the multiple equivalent complex commands locally if they
wanted to validate changes to the npm configuration in advance of pushing, or troubleshoot a problem flagged by the
workflow.
We already have a Task-based version of the workflow and accompanying tasks, which are generally preferred for Arduino
tooling projects and especially appropriate for this specific project which is devoted to Task:
https://github.com/arduino/tooling-project-assets/blob/main/workflow-templates/check-npm-task.md
Those standardized assets are hereby used in this repository, moving the commands from the workflow to an npm:validate
task and switching the workflow to using that task.
The ts:validate task validates the repository's TypeScript configuration files against their JSON schema.
The tsconfig.json file contains comments, which are not supported by the version of the JSON schema validator tool in
use. The comments are removed by the task and the cleaned data written to a temporary file.
Previously, the wrong taskfile variable was used in the command that generates the filename of that temporary file, resulting in it having a different filename than intended. This has no functional effect on the validation results, which are solely based on the content of the file and not its name, however, it makes the validation results confusing to the humans reading the logs.
For example, previously a successful validation produced output with this form:
/tmp/ts-validate-yVxglvosn8/tsconfig-schema-yviXnDDlRy.json valid
The change to using the correct taskfile variable in the command made here results in output in the expected form:
/tmp/ts-validate-yVxglvosn8/tsconfig.json valid
The ajv-cli command line tool is used for validating data files against their JSON schema.
In general, it is preferable, and for some schemas even mandatory, to use the modern versions of ajv-cli. However,
support for the "Draft 4" schema specification was dropped in ajv-cli version 4.0.0. So when working with JSON schemas
that specify that draft, it is necessary to use ajv-cli 3.3.0, the last compatible version.
This means that this project has dependencies on multiple versions of ajv-cli.
The ajv-cli package is distributed via the npm registry. Unfortunately, npm does not have support for managing multiple
versions of a binary dependency.
npm's "alias" feature seems ideal:
https://docs.npmjs.com/cli/v8/commands/npm-install
> npm install <alias>@npm:<name>:
> Install a package under a custom alias. Allows multiple versions of a same-name package side-by-side
However, this is only suitable for code dependencies. There is no special handling of providing access via npx to the
binary of the aliased package.
So the only reasonable option is to manage only one of the two tool versions. This should be a simple matter of
specifying the version of the package in the commands using the unmanaged tool, as is already done
(e.g., `npx ajv-cli@3.3.0`):
https://docs.npmjs.com/cli/v8/commands/npx#description
> Package names with a specifier will only be considered a match if they have the exact same name and version as the
> local dependency.
Unfortunately, a bug in npm causes the version specifier to be ignored when the managed version is already installed.
This bug is reportedly fixed, but the fix is not in a Node.js release yet, and even after that it may take us quite some
time to upgrade to the version of Node.js which contains the fix.
The workaround implemented here is to run the unmanaged tool command from outside the project folder.
Previously, the tools used by the "Check TypeScript Configuration" workflow were installed using `npm install --global`
commands. Although unnecessary, it was acceptable while these installations were only done in the isolated runner
machine the workflow runs in. Now that the commands are moved to a task, to enable the validation operation to be ran
locally on contributor machines, automatically making global installations of tools is unacceptable.
The task is hereby changed to using npx to run the tools, which avoids any global impact on local machines the task runs
on.
The Task task runner tool uses an integrated Bash shell interpreter to allow the use of standard POSIX/Bash syntax and
built-in utilities while providing cross-platform support for users who have made the poor choice of the non-standard
Windows cmd or PowerShell shells.
While this is a nice feature of Task, distinguishing it from other alternatives, it is still often necessary to use
non-built-in utilities. So use of a Bash shell is a requirement to run our tasks even the developers using Windows.
Although high quality Bash shells for Windows such as the Git Bash included as part of the Git for Windows installation
generally provide a seamless experience, there is the occasional "gotcha".
Relative paths work the same for POSIX and Windows, but absolute POSIX format paths may not. These paths are handled as
expected by Task's integrated shell interpreter and by Bash, but when these paths are used outside that context they may
no longer be correct.
For example:
```
foo:
cmds:
- mkdir "/tmp/posix-path-test-dir"
- touch "/tmp/posix-path-test-dir/posix-path-test-file"
- ls "/tmp/posix-path-test-dir"
- cd "/tmp/posix-path-test-dir"
```
The first three commands work as expected, but the `cd` command fails:
```
$ task foo
task: [foo] mkdir "/tmp/posix-path-test-dir"
task: [foo] touch "/tmp/posix-path-test-dir/posix-path-test-file"
task: [foo] ls "/tmp/posix-path-test-dir"
posix-path-test-file
task: [foo] cd "/tmp/posix-path-test-dir"
task: Failed to run task "foo": exit status 1
```
The POSIX format `/tmp` is actually set to the system temporary directory:
```
$ cygpath -w "/tmp/posix-path-test-dir"
C:\Users\per\AppData\Local\Temp\posix-path-test-dir
```
But if treated as a Windows format path, this would be the `tmp` subfolder of the root of the current drive (e.g.,
`C:\tmp`). Even though the command was run from Git Bash, which provides a `cd` that handles this absolute path
perfectly, when run from Task the Windows native `cd` is used instead.
This resulted in several of the data file validation tasks, which download the JSON schema to the temporary folder, to
fail when ran on Windows.
The solution is to convert these absolute paths, the occurrence of which are relatively rare in our tasks, to Windows
format (which is handled fine by both the integrated, Bash, and external applications) when the task is ran on a Windows
machine. The cygpath utility provides this capability:
```
$ task foo
task: [foo] mkdir "C:\Users\per\AppData\Local\Temp/posix-path-test-dir"
task: [foo] touch "C:\Users\per\AppData\Local\Temp/posix-path-test-dir/posix-path-test-file"
task: [foo] ls "C:\Users\per\AppData\Local\Temp/posix-path-test-dir"
posix-path-test-file
task: [foo] cd "C:\Users\per\AppData\Local\Temp/posix-path-test-dir"
```
In addition to helping others use Task in their GitHub Actions workflows, this project uses Task for its own development.
Most of the repository's GitHub Actions workflows use the approach of executing a task. This approach allows maintaining
a single set of commands that can easily be used locally by contributors in addition to their use by the CI system.
The "Check TypeScript Configuration" workflow was an exception to this pattern, using commands directly coded into the
workflow file. This meant the contributor would need to study the workflow and run the multiple equivalent complex
commands locally if they wanted to validate changes to the TypeScript configuration in advance of pushing, or
troubleshoot a problem flagged by the workflow.
These commands are hereby moved to a ts:validate task and the workflow switched to using that task.
This is intended to be a straight across transfer of the existing commands (or their equivalents) to the task.
A task and GitHub Actions workflow are provided here for checking the license types of npm-managed project dependencies.
On every push and pull request that affects relevant files, the CI workflow will check:
- If the dependency licenses cache is up to date
- If any of the project's dependencies have an unapproved license type.
Approval can be based on:
- Universally allowed license type
- Individual dependency
I originally set out to establish some logical order to the tasks on some indecipherable criteria. In the end, it
resulted in a jumble.
Alphabetical order is completely objective and it results in a fairly logical order in the end due to the the use of
prefixes on the task names according to their domain. The exception is that the convenience "umbrella" tasks have been
left at the top (in their own alphabetical order) so that they will have the maximum visibility as the most useful tasks.
Even that is not completely inconsistent, since these don't have a namespace prefix, and so might be imagined to have a
null global namespace prefix that would be sorted first.
The `fix` task provides a standardized interface for the developer to run all available automated correction processes.
Even though some such processes will likely be irrelevant at any given time, it can be more convenient, and often more
efficient, for a casual contributor to just run the "umbrella" task rather than inspecting the available tasks and
running each of the relevant ones individually.
This will make it easier for the maintainers to sync fixes and improvements in either direction between the upstream
"template" assets and their installation in this repository.
`npx --call` uses the native shell. The use of Bash code in the link check command passed to npx made it incompatible with Windows, even when the task was run from Bash.
After quite some struggles, I decided that it's simply too difficult to use npx for this application on Windows. The
workaround is to abandon the use of npx for this task on Windows and instead require Windows users to have
markdown-link-check installed and in PATH, providing a helpful error message in the event it is not.
Generated documentation content may contain links, and it is just as important to check these. This project does not
contain any generated documentation at the moment, so the generation task is left empty, but this is a sync with the
"template" task which is intended to be applicable to any project.
The markdownlint tool used to check for problems or style inconsistencies in the repository's Markdown content provides
an automatic correction capability for certain rules.
A dedicated task has been added for installing the project's Python dependencies via Poetry, and this should be used
throughout the taskfile in place of redundant direct commands.
On every push and pull request that affects relevant files, and periodically, check the repository's Markdown files for
problems:
- Use markdownlint to check for common problems and formatting.
- Use markdown-link-check to check for broken links.
Arduino's Markdown style is defined by the `.markdownlint.yml` file.
In the event the repository contains externally maintained Markdown files, markdownlint can be configured to ignore them
via a `.markdownlintignore` file:
https://github.com/igorshubovych/markdownlint-cli#ignoring-files
markdown-link-check is configured via the `.markdown-link-check.json` file:
https://github.com/tcort/markdown-link-check#config-file-format
On every push, pull request, and periodically, use the codespell-project/actions-codespell action to check for commonly
misspelled words.
In the event of a false positive, the problematic word should be added, in all lowercase, to the ignore-words-list field
of ./.codespellrc. Regardless of the case of the word in the false positive, it must be in all lowercase in the ignore
list. The ignore list is comma-separated with no spaces.
