# gotest.tools

A collection of packages to augment `testing` and support common patterns.

[![GoDoc](https://godoc.org/gotest.tools?status.svg)](https://pkg.go.dev/gotest.tools/v3/?tab=subdirectories)
[![CircleCI](https://circleci.com/gh/gotestyourself/gotest.tools/tree/main.svg?style=shield)](https://circleci.com/gh/gotestyourself/gotest.tools/tree/main)
[![Go Reportcard](https://goreportcard.com/badge/gotest.tools)](https://goreportcard.com/report/gotest.tools)

## Usage

With Go modules enabled (go1.11+)

```
$ go get gotest.tools/v3
```

```
import "gotest.tools/v3/assert"
```

To use `gotest.tools` with an older version of Go that does not understand Go
module paths pin to version `v2.3.0`.


## Packages

* [assert](http://pkg.go.dev/gotest.tools/v3/assert) -
  compare values and fail the test when a comparison fails
* [env](http://pkg.go.dev/gotest.tools/v3/env) -
  test code which uses environment variables
* [fs](http://pkg.go.dev/gotest.tools/v3/fs) -
  create temporary files and compare a filesystem tree to an expected value
* [golden](http://pkg.go.dev/gotest.tools/v3/golden) -
  compare large multi-line strings against values frozen in golden files
* [icmd](http://pkg.go.dev/gotest.tools/v3/icmd) -
  execute binaries and test the output
* [poll](http://pkg.go.dev/gotest.tools/v3/poll) -
  test asynchronous code by polling until a desired state is reached
* [skip](http://pkg.go.dev/gotest.tools/v3/skip) -
  skip a test and print the source code of the condition used to skip the test

## Related

* [gotest.tools/gotestsum](https://github.com/gotestyourself/gotestsum) -
  go test runner with custom output
* [go testing patterns](https://github.com/gotestyourself/gotest.tools/wiki/Go-Testing-Patterns) -
  zero-dependency patterns for organizing test cases
* [test doubles and patching](https://github.com/gotestyourself/gotest.tools/wiki/Test-Doubles-And-Patching) -
  zero-dependency test doubles (fakes, spies, stubs, and mocks) and monkey patching patterns

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).


gotestyourself/gotest.tools

A collection of packages to augment the go testing package and support common patterns.

Go (aka Golang) is an open source programming language that makes it easy to build performant, cross-platform applications

Apache License 2.0

viant/endly

End to end functional test and automation framework

# gotestsum

`gotestsum` runs tests using `go test -json`, prints formatted test output, and a summary of the test run.
It is designed to work well for both local development, and for automation like CI.

## Install

Download a binary from [releases](https://github.com/gotestyourself/gotestsum/releases), or build from
source with `go install gotest.tools/gotestsum@latest`. With `go` version before 1.17, use `go get gotest.tools/gotestsum`.

## Documentation

**Core features**
- Change the [test output format](#output-format), from compact to verbose with color highlighting.
- Print a [summary](#summary) of the test run after running all the tests.
- Use any [`go test` flag](#custom-go-test-command),
  run a script with [`--raw-command`](#custom-go-test-command),
  or [run a compiled test binary](#executing-a-compiled-test-binary).

**CI and Automation**
- [`--junitfile`](#junit-xml-output) - write a JUnit XML file for integration with CI systems.
- [`--jsonfile`](#json-file-output) - write all the [test2json](https://pkg.go.dev/cmd/test2json) input received by `gotestsum` to a file. The file
  can be used as input to [`gotestsum tool slowest`](#finding-and-skipping-slow-tests), or as a way to
  store the full verbose output of tests when less verbose output is printed to stdout using a compact [`--format`](#output-format).
- [`--rerun-fails`](#re-running-failed-tests) - run failed (possibly flaky) tests again to avoid re-running the
  entire suite. Re-running individual tests can save significant time when working with flaky test suites.

**Local Development**
- [`--watch`](#run-tests-when-a-file-is-saved) - every time a `.go` file is saved run the tests for the package that changed.
- [`--post-run-command`](#post-run-command) - run a command after the tests, can be used for desktop notification of the test run.
- [`gotestsum tool slowest`](#finding-and-skipping-slow-tests) - find the slowest tests, or automatically update the source code of
  the slowest tests to add a conditional `t.Skip` statements. This statement allows you to skip the slowest tests using `gotestsum -- -short ./...`.


### Output Format

The `--format` flag or `GOTESTSUM_FORMAT` environment variable set the format that
is used to print the test names, and possibly test output, as the tests run. Most
outputs use color to highlight pass, fail, or skip.

The `--format-hivis` flag changes the icons used by `pkgname` formats to higher
visiblity unicode characters.

Commonly used formats (see `--help` for a full list):

 * `dots` - print a character for each test.
 * `pkgname` (default) - print a line for each package.
 * `testname` - print a line for each test and package.
 * `standard-quiet` - the standard `go test` format.
 * `standard-verbose` - the standard `go test -v` format.

Have an idea for a new format?
Please [share it on github](https://github.com/gotestyourself/gotestsum/issues/new)!

#### Demo

A demonstration of three `--format` options.

![Demo](https://user-images.githubusercontent.com/442180/182284939-e08a0aa5-4504-4e30-9e88-207ef47f4537.gif)
<br /><sup>[Source](https://github.com/gotestyourself/gotestsum/tree/readme-demo/scripts)</sup>

### Summary

Following the formatted output is a summary of the test run. The summary includes:

 * The test output, and elapsed time, for any test that fails or is skipped.
 * The build errors for any package that fails to build.
 * A `DONE` line with a count of tests run, tests skipped, tests failed, package build errors,
   and the elapsed time including time to build.

   ```
   DONE 101 tests[, 3 skipped][, 2 failures][, 1 error] in 0.103s
   ```

To hide parts of the summary use `--hide-summary section`.


**Example: hide skipped tests in the summary**
```
gotestsum --hide-summary=skipped
```

**Example: hide everything except the DONE line**
```
gotestsum --hide-summary=skipped,failed,errors,output
# or
gotestsum --hide-summary=all
```

**Example: hide test output in the summary, only print names of failed and skipped tests
and errors**
```
gotestsum --hide-summary=output
```

### JUnit XML output

When the `--junitfile` flag or `GOTESTSUM_JUNITFILE` environment variable are set
to a file path, `gotestsum` will write a test report, in JUnit XML format, to the file.
This file can be used to integrate with CI systems.

```
gotestsum --junitfile unit-tests.xml
```

If the package names in the `testsuite.name` or `testcase.classname` fields do not
work with your CI system these values can be customized using the
`--junitfile-testsuite-name`, or `--junitfile-testcase-classname` flags. These flags
accept the following values:

* `short` - the base name of the package (the single term specified by the
  package statement).
* `relative` - a package path relative to the root of the repository
* `full` - the full package path (default)


Note: If Go is not installed, or the `go` binary is not in `PATH`, the `GOVERSION`
environment variable can be set to remove the "failed to lookup go version for junit xml"
warning.

### JSON file output

When the `--jsonfile` flag or `GOTESTSUM_JSONFILE` environment variable are set
to a file path, `gotestsum` will write a line-delimited JSON file with all the
[test2json](https://golang.org/cmd/test2json/#hdr-Output_Format)
output that was written by `go test -json`. This file can be used to compare test
runs, or find flaky tests.

```
gotestsum --jsonfile test-output.log
```

### Post Run Command

The `--post-run-command` flag may be used to execute a command after the
test run has completed. The binary will be run with the following environment
variables set:

```
GOTESTSUM_FORMAT        # gotestsum format (ex: short)
GOTESTSUM_JSONFILE      # path to the jsonfile, empty if no file path was given
GOTESTSUM_JUNITFILE     # path to the junit.xml file, empty if no file path was given
TESTS_ERRORS            # number of errors
TESTS_FAILED            # number of failed tests
TESTS_SKIPPED           # number of skipped tests
TESTS_TOTAL             # number of tests run
```

To get more details about the test run, such as failure messages or the full list of failed
tests, run `gotestsum` with either a `--jsonfile` or `--junitfile` and parse the
file from the post-run-command. The
[gotestsum/testjson](https://pkg.go.dev/gotest.tools/gotestsum/testjson?tab=doc)
package may be used to parse the JSON file output.

**Example: desktop notifications**

First install the example notification command with `go get gotest.tools/gotestsum/contrib/notify`.
The command will be downloaded to `$GOPATH/bin` as `notify`. Note that this
example `notify` command only works on macOS with
[terminal-notifer](https://github.com/julienXX/terminal-notifier) installed.

```
gotestsum --post-run-command notify
```

**Example: command with flags**

Possitional arguments or command line flags can be passed to the `--post-run-command` by
quoting the whole command.

```
gotestsum --post-run-command "notify me --date"
```

### Re-running failed tests

When the `--rerun-fails` flag is set, `gotestsum` will re-run any failed tests.
The tests will be re-run until each passes once, or the number of attempts
exceeds the maximum attempts. Maximum attempts defaults to 2, and can be changed
with `--rerun-fails=n`.

To avoid re-running tests when there are real failures, the re-run will be
skipped when there are too many test failures. By default this value is 10, and
can be changed with `--rerun-fails-max-failures=n`.

Note that using `--rerun-fails` may require the use of other flags, depending on
how you specify args to `go test`:

* when used with `--raw-command` the re-run will pass additional arguments to
  the command. The first arg is a `-test.run` flag with a regex that matches the test to re-run,
  and second is the name of a go package. These additional args can be passed to `go test`,
  or a test binary.
* when used with any `go test` args (anything after `--` on the command line), the list of
  packages to test must be specified as a space separated list using the `--packages` arg.

  **Example**

  ```
  gotestsum --rerun-fails --packages="./..." -- -count=2
  ```

* if any of the `go test` args should be passed to the test binary, instead of
  `go test` itself, the `-args` flag must be used to separate the two groups of
  arguments. `-args` is a special flag that is understood by `go test` to indicate
  that any following args should be passed directly to the test binary.

  **Example**

  ```
  gotestsum --rerun-fails --packages="./..." -- -count=2 -args -update-golden
  ```


### Custom `go test` command

By default `gotestsum` runs tests using the command `go test -json ./...`. You
can change the command with positional arguments after a `--`. You can change just the
test directory value (which defaults to `./...`) by setting the `TEST_DIRECTORY`
environment variable.

You can use `--debug` to echo the command before it is run.

**Example: set build tags**
```
gotestsum -- -tags=integration ./...
```

**Example: run tests in a single package**
```
gotestsum -- ./io/http
```

**Example: enable coverage**
```
gotestsum -- -coverprofile=cover.out ./...
```

**Example: run a script instead of `go test`**
```
gotestsum --raw-command -- ./scripts/run_tests.sh
```

Note: when using `--raw-command`, the script must follow a few rules about
stdout and stderr output:

* The stdout produced by the script must only contain the `test2json` output, or
  `gotestsum` will fail. If it isn't possible to change the script to avoid
  non-JSON output, you can use `--ignore-non-json-output-lines` (added in version 1.7.0)
  to ignore non-JSON lines and write them to `gotestsum`'s stderr instead.
* Any stderr produced by the script will be considered an error (this behaviour
  is necessary because package build errors are only reported by writting to
  stderr, not the `test2json` stdout). Any stderr produced by tests is not
  considered an error (it will be in the `test2json` stdout).

**Example: accept intput from stdin**
```
cat out.json | gotestsum --raw-command -- cat
```

**Example: run tests with profiling enabled**

Using a `profile.sh` script like this:

```sh
#!/usr/bin/env bash
set -eu

for pkg in $(go list "$@"); do
    dir="$(go list -f '{{ .Dir }}' $pkg)"
    go test -json -cpuprofile="$dir/cpu.profile" "$pkg"
done
```

You can run:
```
gotestsum --raw-command ./profile.sh ./...
```

**Example: using `TEST_DIRECTORY`**
```
TEST_DIRECTORY=./io/http gotestsum
```

### Executing a compiled test binary

`gotestsum` supports executing a compiled test binary (created with `go test -c`) by running
it as a custom command.

The `-json` flag is handled by `go test` itself, it is not available when using a
compiled test binary, so `go tool test2json` must be used to get the output
that `gotestsum` expects.

**Example: running `./binary.test`**

```
gotestsum --raw-command -- go tool test2json -t -p pkgname ./binary.test -test.v
```

`pkgname` is the name of the package being tested, it will show up in the test
output. `./binary.test` is the path to the compiled test binary. The `-test.v`
must be included so that `go tool test2json` receives all the output.

To execute a test binary without installing Go, see
[running without go](./.project/docs/running-without-go.md).


### Finding and skipping slow tests

`gotestsum tool slowest` reads [test2json output][testjson],
from a file or stdin, and prints the names and elapsed time of slow tests.
The tests are sorted from slowest to fastest.

`gotestsum tool slowest` can also rewrite the source of tests slower than the
threshold, making it possible to optionally skip them.

The [test2json output][testjson] can be created with `gotestsum --jsonfile` or `go test -json`.

See `gotestsum tool slowest --help`.

**Example: printing a list of tests slower than 500 milliseconds**

```
$ gotestsum --format dots --jsonfile json.log
[.]····↷··↷·
$ gotestsum tool slowest --jsonfile json.log --threshold 500ms
gotest.tools/example TestSomething 1.34s
gotest.tools/example TestSomethingElse 810ms
```

**Example: skipping slow tests with `go test --short`**

Any test slower than 200 milliseconds will be modified to add:

```go
if testing.Short() {
    t.Skip("too slow for testing.Short")
}
```

```sh
go test -json -short ./... | gotestsum tool slowest --skip-stmt "testing.Short" --threshold 200ms
```

Use `git diff` to see the file changes.
The next time tests are run using `--short` all the slow tests will be skipped.

[testjson]: https://golang.org/cmd/test2json/


### Run tests when a file is saved 

When the `--watch` flag is set, `gotestsum` will watch directories using
[file system notifications](https://pkg.go.dev/github.com/fsnotify/fsnotify).
When a Go file in one of those directories is modified, `gotestsum` will run the
tests for the package that contains the changed file. By default all
directories under the current
directory with at least one `.go` file will be watched.
Use the `--packages` flag to specify a different list.

If `--watch` is used with a command line that includes the name of one or more
packages as command line arguments (ex: `gotestsum --watch -- ./...` or
`gotestsum --watch -- ./extrapkg`), the
tests in those packages will also be run when any file changes.

With the `--watch-chdir` flag, `gotestsum` will change the working directory
to the directory with the modified file before running tests. Changing the
directory is primarily useful when the project contains multiple Go modules.
Without this flag, `go test` will refuse to run tests for any package outside
of the main Go module.

While in watch mode, pressing some keys will perform an action:

* `r` will run tests for the previous event.
  Added in version 1.6.1.
* `u` will run tests for the previous event, with the `-update` flag added.
  Many [golden](https://gotest.tools/v3/golden) packages use this flag to automatically
  update expected values of tests.
  Added in version 1.8.1.
* `d` will run tests for the previous event using `dlv test`, allowing you to 
  debug a test failure using [delve]. A breakpoint will automatically be added at
  the first line of any tests which failed in the previous run. Additional
  breakpoints can be added with [`runtime.Breakpoint`](https://golang.org/pkg/runtime/#Breakpoint)
  or by using the delve command prompt.
  Added in version 1.6.1.
* `a` will run tests for all packages, by using `./...` as the package selector.
  Added in version 1.7.0.
* `l` will scan the directory list again, and if there are any new directories
  which contain a file with a `.go` extension, they will be added to the watch
  list.
  Added in version 1.7.0.

Note that [delve] must be installed in order to use debug (`d`).

[delve]: https://github.com/go-delve/delve

**Example: run tests for a package when any file in that package is saved**
```
gotestsum --watch --format testname
```

## Development

[![Godoc](https://godoc.org/gotest.tools/gotestsum?status.svg)](https://pkg.go.dev/gotest.tools/gotestsum?tab=subdirectories)
[![CircleCI](https://circleci.com/gh/gotestyourself/gotestsum/tree/main.svg?style=shield)](https://circleci.com/gh/gotestyourself/gotestsum/tree/main)
[![Go Recipes](https://raw.githubusercontent.com/nikolaydubina/go-recipes/main/badge.svg?raw=true)](https://github.com/nikolaydubina/go-recipes)
[![Go Reportcard](https://goreportcard.com/badge/gotest.tools/gotestsum)](https://goreportcard.com/report/gotest.tools/gotestsum)


Pull requests and bug reports are welcome! Please open an issue first for any
big changes.

## Thanks

This package is heavily influenced by the [pytest](https://docs.pytest.org) test runner for `python`.


gotestyourself/gotestsum

'go test' runner with output optimized for humans, JUnit XML for CI integration, and a summary of the test results.