# Visual Regression Test Runner

This directory contains both the code for the Renderer's Visual Regression Test
Runner and a directory of certified snapshot images
(`visual-regression/certified-snapshots`) for each of the defined Visual
Regression Test Case Snapshots (Snapshots).

Within the `certified-snapshots` directory are subdirectories defining
which browser and runtime environment (RUNTIME_ENV environment variable) was
used to generate their contained snapshots in the format `${browser}-${env}`.

The supported runtime environments are `ci` (generated from a Linux-based Docker
container) and `local` (generated by the local machine environment).

`ci` snapshots are checked into the repo and are the basis of a GitHub
Actions PR status check.

`local` snapshots are never checked into the repo and are used strictly to speed
up local development. See instructions below.

NOTE: Currently these tests only run the Chromium web browser as a baseline.
Other browser support may come in the future.

## How to run

```
Visual Regression Test Runner
Options:
      --help       Show help                                           [boolean]
      --version    Show version number                                 [boolean]
  -c, --capture    Capture new snapshots              [boolean] [default: false]
  -o, --overwrite  Overwrite existing snapshots (--capture must also be set)
                                                      [boolean] [default: false]
  -v, --verbose    Verbose output                     [boolean] [default: false]
  -s, --skipBuild  Skip building renderer and examples[boolean] [default: false]
  -p, --port       Port to serve examples on           [number] [default: 50535]
  -i, --ci         Run in docker container with `ci` runtime environment
                                                      [boolean] [default: false]
  -f, --filter     Tests to run ("*" wildcard pattern)   [string] [default: "*"]
```

The test runner may be launched in local mode with:

```
pnpm test:visual
```

To run the tests in a Docker container with the `ci` runtime environment, use `--ci` mode:

```
pnpm test:visual --ci
```

NOTE: For this to work, you must have Docker installed and have built the
Visual Regression Docker Image. See [DOCKER.md](./DOCKER.md) for more info.

### Comparison Mode (Default)

By default, the runner will build the Renderer, then the Example Tests, and then
serve/launch the tests in a headless browser. The actual screenshot output from
the headless browser for each defined Visual Regression Snapshot will then be
compared pixel-by-pixel to the certified expected snapshot images. If a
difference is detected, the test will fail.

To make it easy to see what went wrong in any failed Snapshot, three images will
be saved to the `visual-regression/failed-results`: the certified expected
snapshot image, the actual snapshot image, and a difference between the two of
them. As a protection, a Git hook exists that will prevent a commit if this
directory contains any failed result files. The failed results are cleared
before every comparison run.

If any test fails the exit code of the test runner will be `1` to indicate there
was a failure. Otherwise it will be `0`.

### Capture Mode

In order to capture new Snapshots (or overwrite existing ones), you need to run
the Visual Regression Test Runner in Capture mode:

```
pnpm test:visual [--ci] --capture
```

This will do everything that the Comparison mode does, but skip the actual
comparing and instead capture and save the Snapshot image data files to files
in the `visual-regression/certified-snapshots` directory.

As a safety feature, by default, Capture mode will not overwrite any existing
certified snapshot files. So in order to capture new ones it is recommended
to delete the specific certified snapshot files you'd like to update before
running Capture mode. If you are sure you'd like to overwrite all of the
certified snapshots, you can add the `--overwrite` CLI argument to the command.

### Local Snapshot Testing

When developing locally, running the tests in Docker CI mode can be significantly
slower than than running them on the local runtime environment. Local snapshots
are never checked into the repository, however you can utilize local snapshots
to speed up the time it takes to verify if your changes cause any regressions.

1. After creating a new branch for a new feature or bug fix, capture a new set
   of local snapshots by running: `pnpm test:visual --capture --overwrite`
2. Implement the new feature or bug fix.
3. Before pushing new commits to the remote PR branch, compare your changes
   to the snapshots you took prior to starting your PR: `pnpm test:visual`

## GitHub Actions Workflow

The Visual Regression Tests run as a status check on every pull request update
via the GitHub Actions workflow defined in `.github/workflows/tests.yml`. The tests
that run are compared with the snapshots in the `certified-snapshots/chromium-ci`
directory.

When tests failed, the failure results that you normally find in
`visual-regression/failed-results` are uploaded to the workflow run as a zip file
artifact named **failed-results**. See [Where does the upload go?](https://github.com/actions/upload-artifact#where-does-the-upload-go)
for more on how to find it.

## How to define Snapshots

The Snapshots themselves are defined in the individual Example Tests located in the
`examples/tests` directory. Note that not all Example Tests need to define Snapshots.
For an Example Test to define Snapshots, it must export an `automation()`
function. Here's an example of one from the `alpha` Example Test:

```ts
export async function automation(settings: ExampleSettings) {
  // Launch the test
  await test(settings);
  // Take/define a snapshot
  await settings.snapshot();
}
```

This method is called only when the Visual Regression Tests are run. Here the
Example Test only defines one Snapshot. It first runs the Example Test's
renderer code by calling `test()` (defined later in the Example Test's code),
and then takes a single snapshot by calling `settings.snapshot()`. When the
Visual Regression Tests are run, this Snapshot will be given the name
`alpha-1` since it defines the 1st (and only) Snapshot of the `alpha` Example
Test. Addtional snapshots can be defined by calling `settings.snapshot()`
additional times, while of course making changes to the Renderer state in
between calls.

A name may be optionally provided in the snapshot call:

```typescript
settings.snapshot({ name: 'myname' });
```

This name will be appended to the name of the Example Test. For example, if
run in the `alpha` Example Test, the name of Snapshot will be `alpha_myname-1`.
The same name may be used multiple times.

Example Tests that utilize the `PageContainer` class to define separate pages
of static content may use the `pageContainer.snapshotPages()` helper method
to automatically take snapshots of each of the pages defined in the container.
Here's an example from the `text-rotation` Example Test:

```ts
export async function automation(settings: ExampleSettings) {
  // Snapshot all the pages (`await test()` resolves to a PageContainer instance)
  await (await test(settings)).snapshotPages();
}
```

### Defining Smaller Snapshots

By default, when calling `settings.snapshot()` a snapshot will be created of
the entire app/testRoot area at 1920x1080 (downscaled to canvas/PNG at 1280x720).
Large snapshots like the default require more processing to load and do a diff
on. If your test only requires a small snapshot area you can communicate this
to the Visual Regression Test Runner in two ways:

**Method 1:** Set the dimensions of the testRoot.

```ts
// Do this in the actual test itself (not the automation function) for the best
// results.
export default function test(settings: ExampleSettings) {
  const { testRoot } = settings;
  // Set a smaller snapshot area
  testRoot.width = 200;
  testRoot.height = 200;

  // Set a color on the test root so its more obvious when running the test in
  // the browser what the snapshot area is.
  testRoot.color = 0xffffffff;
}
```

**Method 2:** Provide a `clip` rectangle prop to the `settings.snapshot()` method.

```ts
export async function automation(settings: ExampleSettings) {
  // Launch the test
  await test(settings);
  // Take/define a snapshot
  await settings.snapshot({
    clip: {
      x: 0,
      y: 0,
      width: 200,
      height: 200,
    },
  });
}
```

Method 2 allows more flexibility in how snapshots are defined but should only
be used if Method 1 does not satisfy the requirements for the snapshots to be
taken.

### Math.random()

In order to allow for consistent snapshots when random numbers are convenient in
writing tests, Example Tests that are run by the Visual Regression Test Runner
receive a consistent constant [seeded implementation](https://github.com/stdlib-js/random-base-mt19937)
of `Math.random()`.

This means that for each Example Test that is run, `Math.random()` will
produce the same sequence of random numbers.

`Math.random()` operates as it normally does when the Example Tests are run
directly in the browser (without the VRT Runnner).
