Skip to main content
Documentation menu

Testing with cabin test

cabin test is a small wrapper around the existing build pipeline: it builds the selected test targets, runs each linked executable in deterministic order, and reports a summary.

It is intentionally not a testing framework. It does no test- case discovery inside binaries and parses no framework-specific output. The unit of execution is the entire test executable - its exit status decides pass / fail.

Declaring a test target

[target.demo_test]
type = "test"
sources = ["tests/lib_test.cc"]
deps = ["demo"]

The full test syntax is documented in docs/targets.md.

Running tests

cabin test                           # every test in the default selection
cabin test --workspace               # every test in every workspace member
cabin test -p demo                   # only demo's tests
cabin test --test demo_test          # only the named test target
cabin test --test a --test b         # several named test targets
cabin test --release                 # compile with the release profile
cabin test --features simd          # forward features to the test build
cabin test -- --gtest_filter='Suite.*'  # pass arguments to every test executable
cabin test --test-jobs 4             # run up to four test executables at once
cabin test --test-timeout 60         # kill a test still running after 60 seconds
cabin test --junit report.xml        # also write a JUnit XML report

--test <NAME> runs individual test targets, mirroring cargo test --test <name>. The flag may be repeated; repeated names are deduplicated. Each requested name must match a test target declared by a selected package - an unknown name (or a name that matches a target of another kind) is an error, even under --allow-no-tests. Every match across the selected packages runs, so two workspace members may share a test name. Package selection composes with --test: names are looked up in the selected packages only.

A test target with required-features follows the general gating rules: the no---test enumeration skips it when its features are not enabled (it counts as filtered out in the summary), while naming it with --test <NAME> is a hard error that spells out the missing features. When every declared test is gated off, cabin test fails with an error naming the gated targets (or, under --allow-no-tests, reports the filtered-out count) so the outcome stays distinguishable from a package with no tests at all.

cabin test shares its core flags with cabin build: -j / --jobs, --profile, --release, --features, --no-default-features, --all-features, --locked, --frozen, --no-patches, the toolchain overrides (--cc / --cxx / --ar / --compiler-wrapper), the workspace-selection bundle (--workspace / -p / --default-members / --exclude), and the index/cache locations (--index-path / --index-url / --cache-dir). With --target naming another machine, it runs the tests through the triple’s configured [runner], or builds them and fails without running them when none is configured.

--allow-no-tests opts the user out of the empty-selection error: by default, cabin test exits with an error when the selected packages declare no test target, so CI does not silently pass when tests have not been added yet. Pass --allow-no-tests for cases where empty is expected.

Passing arguments to tests

Everything after -- is passed to every selected test executable as its argument list, in order and without a shell: an empty argument, an argument with spaces or quotes, a second --, and (on Unix) bytes that are not UTF-8 reach the test unchanged. On Windows, where a process receives one command line, the list is quoted so that the C runtime’s parsing recovers it; a narrow main receives that list in the ANSI code page, so characters outside it reach only a wmain. The test framework applies its own filters and options:

cabin test --test unit_test -- --gtest_filter='Parser.*' --gtest_repeat=3
cabin test -- --success "[tag]"           # Catch2

Without --, a bare argument is rejected rather than taken as a test filter. The arguments go to every selected executable, so combine a framework-specific option with --test <NAME> when the selection mixes frameworks.

Concurrency and timeouts

--test-jobs <N> runs up to N test executables at once; the default is 1, one after another. Tests run in their package’s directory and may share fixture files there, so raise it once the tests are independent of each other. It is separate from -j / --jobs, which limits the build exactly as for cabin build, including the CABIN_BUILD_JOBS and [build] jobs fallbacks. Every selected test runs, whatever the others do.

--test-timeout <SECONDS> kills a test executable still running after that many seconds and reports it as failed. Only the test process itself is killed; processes it started are not tracked. Once a test exits, Cabin waits at most a second for its output to close: if a process the test started still holds its stdout or stderr, Cabin stops reading, and output written after that is not shown.

Interrupting cabin test (Ctrl-C) stops it without a summary. The interrupt also reaches the running tests, which share the terminal with Cabin.

Output and exit status

The status lines mirror cargo test’s shape. A run prints the running N tests header, one result line per executable as it finishes, and a summary line:

running 2 tests
test <pkg>:<target> ... ok
test <pkg>:<target> ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s

Cabin has no ignore or benchmark mechanism, so ignored and measured are constant zeros - they keep the summary line shaped exactly like cargo test’s. filtered out counts the test targets in the selected packages that the invocation deselected via --test <NAME> or skipped because their required-features are not enabled. finished in is the wall-clock time of the test run (the build is not included).

A test executable’s stdout / stderr appear under ---- stdout: <pkg>:<target> ---- / ---- stderr: <pkg>:<target> ---- headers. With --test-jobs 1 they stream live while the test runs; unlike cargo test, output is not buffered until the end of the run. With more test jobs, each executable’s output is printed in one piece when it finishes, just before its result line, so output from concurrent tests never interleaves. Result lines appear in the order tests finish.

A test that does not pass is reported with the reason:

test <pkg>:<target> ... FAILED (exit 17)
test <pkg>:<target> ... FAILED (terminated by signal 11)
test <pkg>:<target> ... FAILED (timed out after 60s)
test <pkg>:<target> ... FAILED (could not start: Permission denied (os error 13))

Signals are reported on Unix only. A result line ends in (output cut off) when Cabin stopped reading output that a leftover process held open (see timeouts).

If any test fails, cabin test itself exits non-zero. A failures: recap lists the failed test names before the summary, followed by the top-level error on stderr:

failures:
    <pkg>:<target>

test result: FAILED. P passed; F failed; 0 ignored; 0 measured; FO filtered out; finished in T.TTs
error: test failures: F of N test executables failed

JUnit reports

--junit <PATH> writes a JUnit XML report once the tests have run, whether or not they passed; CI services that read JUnit can show it next to the job. Cabin sees test executables, not the test cases inside them, so each executable is one testcase (named after the target, with the package as classname) and each package one testsuite. Each case carries the executable’s run time and its stdout and stderr as system-out and system-err. A test that fails, is terminated by a signal, or times out is a failure whose message is the reason the result line shows (exit 17, timed out after 60s); a test that cannot start is an error. Output is decoded as UTF-8, with replacement characters for invalid bytes and for U+FFFE and U+FFFF, which XML cannot carry. ANSI escapes and the C0 and C1 control characters other than the line feed are dropped, tabs and carriage returns included, and an escape sequence that never ends hides the rest of that stream; the terminal output keeps everything. A test whose output was cut off has the property output-cut-off set to true. No report is written when no test runs.

For a report per test case, pass the test framework its own reporting arguments after --, such as GoogleTest’s --gtest_output=xml:<dir>/, which names each file after its executable; a relative path resolves against each package’s directory, and --test limits the run to the executables that accept the argument.

Working directory and environment

Each test executable runs with its working directory set to the owning package’s manifest directory. Tests that read fixture data from paths relative to the package root therefore see the same files in CI and on a developer’s machine.

The environment the test inherits is the same as cabin test’s own, without the registry credentials (CABIN_REGISTRY_TOKEN, ACTIONS_ID_TOKEN_REQUEST_TOKEN) and plus the deterministic CABIN_* package-execution overlay (manifest dir / path, package name / version, profile, build dir) documented in environment-variables.md. Cabin does not set test-framework-specific variables; if you need a particular variable, set it in the parent process. Standard input is empty: tests that run at the same time cannot share a terminal.

Determinism

Tests start in (package_name, target_name) ascending order, regardless of the order targets appear in cabin.toml, and the failures: recap uses the same order. With --test-jobs 1 each test runs to completion before the next starts.

Dev-dependencies

cabin test activates [dev-dependencies] for the selected packages as real graph edges. A test target may therefore depend on packages declared only under [dev-dependencies]:

[dependencies]
demo = { path = "../demo" }

[dev-dependencies]
"cabin-ports/googletest" = "=1.17.0"

[target.demo_test]
type = "test"
sources = ["tests/lib_test.cc"]
deps = ["demo", "cabin-ports/googletest"]

Every dependency source form works under [dev-dependencies]: versioned registry requirements (resolved and fetched like normal deps, and recorded in the lockfile) and path deps. The unit-test-gtest example links the cabin-ports/googletest registry package from [dev-dependencies] this way.

Only dev-only target kinds (test, example) may list dev dependencies in deps; an ordinary library / executable target referencing one fails with a diagnostic naming the [dev-dependencies] policy, so a production target cannot accidentally link test-only code.

cabin build continues to ignore [dev-dependencies], so ordinary production builds remain unaffected. Dev-dep activation never propagates: a transitive package’s own [dev-dependencies] stay declaration-only even under cabin test.

Lockfile behavior

--locked and --frozen apply to cabin test exactly as they do to cabin build:

  • --locked requires the lockfile to satisfy the resolver’s pick (and the patch / source-replacement state to match);
  • --frozen additionally forbids state-writing side effects (cache population, lockfile mutation).

Because cabin test includes [dev-dependencies] in resolution, projects that have never run cabin test may need one un---locked invocation to add dev-deps to the lockfile.

What cabin test is not

  • It does not run examples. example targets are not selectable from the command line - they reach the build graph only as transitive deps of another selected target.
  • It does not parse GoogleTest / Catch2 / doctest output; its JUnit report has one test case per executable.
  • It does not filter test cases inside an executable itself - --test <NAME> selects whole test targets; arguments after -- let the test framework filter its own cases.

Contributing tests to Cabin

The rules for writing Cabin’s own test suite - environment isolation via the shared cabin() helper, tool-availability gating, host-path and driver-name conventions, and the CI portability boundary - are contributor guidance, not part of the cabin test user surface. They live in crates/AGENTS.md.