Testing with cabin test
cabin test is a small wrapper around the existing build pipeline: it builds the selected test
targets, runs each linked executable in deterministic order, and reports a summary.
It is intentionally not a testing framework. It does no test- case discovery inside binaries and parses no framework-specific output. The unit of execution is the entire test executable - its exit status decides pass / fail.
Declaring a test target
[target.demo_test]
type = "test"
sources = ["tests/lib_test.cc"]
deps = ["demo"]
The full test syntax is documented in docs/targets.md.
Running tests
cabin test # every test in the default selection
cabin test --workspace # every test in every workspace member
cabin test -p demo # only demo's tests
cabin test --test demo_test # only the named test target
cabin test --test a --test b # several named test targets
cabin test --release # compile with the release profile
cabin test --features simd # forward features to the test build
cabin test -- --gtest_filter='Suite.*' # pass arguments to every test executable
cabin test --test-jobs 4 # run up to four test executables at once
cabin test --test-timeout 60 # kill a test still running after 60 seconds
cabin test --junit report.xml # also write a JUnit XML report
--test <NAME> runs individual test targets, mirroring cargo test --test <name>. The flag may
be repeated; repeated names are deduplicated. Each requested name must match a test target
declared by a selected package - an unknown name (or a name that matches a target of another kind)
is an error, even under --allow-no-tests. Every match across the selected packages runs, so two
workspace members may share a test name. Package selection composes with --test: names are looked
up in the selected packages only.
A test target with required-features follows the general
gating rules: the no---test enumeration skips it when its features are not enabled (it counts as
filtered out in the summary), while naming it with --test <NAME> is a hard error that spells
out the missing features. When every declared test is gated off, cabin test fails with an
error naming the gated targets (or, under --allow-no-tests, reports the filtered-out count) so
the outcome stays distinguishable from a package with no tests at all.
cabin test shares its core flags with cabin build: -j / --jobs, --profile, --release,
--features, --no-default-features, --all-features, --locked, --frozen, --no-patches, the
toolchain overrides (--cc / --cxx / --ar / --compiler-wrapper), the workspace-selection
bundle (--workspace / -p / --default-members / --exclude), and the index/cache locations
(--index-path / --index-url / --cache-dir). With
--target naming another machine, it runs the
tests through the triple’s configured [runner], or builds them and fails
without running them when none is configured.
--allow-no-tests opts the user out of the empty-selection error: by default, cabin test exits
with an error when the selected packages declare no test target, so CI does not silently pass when
tests have not been added yet. Pass --allow-no-tests for cases where empty is expected.
Passing arguments to tests
Everything after -- is passed to every selected test executable as its argument list, in order
and without a shell: an empty argument, an argument with spaces or quotes, a second --, and (on
Unix) bytes that are not UTF-8 reach the test unchanged. On Windows, where a process receives one
command line, the list is quoted so that the C runtime’s parsing recovers it; a narrow main
receives that list in the ANSI code page, so characters outside it reach only a wmain. The test
framework applies its own filters and options:
cabin test --test unit_test -- --gtest_filter='Parser.*' --gtest_repeat=3
cabin test -- --success "[tag]" # Catch2
Without --, a bare argument is rejected rather than taken as a test filter. The arguments go to
every selected executable, so combine a framework-specific option with --test <NAME> when the
selection mixes frameworks.
Concurrency and timeouts
--test-jobs <N> runs up to N test executables at once; the default is 1, one after another.
Tests run in their package’s directory and may share fixture files there, so raise it once the
tests are independent of each other. It is separate from -j / --jobs, which limits the build
exactly as for cabin build,
including the CABIN_BUILD_JOBS and [build] jobs fallbacks. Every selected test runs, whatever
the others do.
--test-timeout <SECONDS> kills a test executable still running after that many seconds and
reports it as failed. Only the test process itself is killed; processes it started are not
tracked. Once a test exits, Cabin waits at most a second for its output to close: if a process the
test started still holds its stdout or stderr, Cabin stops reading, and output written after that
is not shown.
Interrupting cabin test (Ctrl-C) stops it without a summary. The interrupt also reaches the
running tests, which share the terminal with Cabin.
Output and exit status
The status lines mirror cargo test’s shape. A run prints the running N tests header, one result
line per executable as it finishes, and a summary line:
running 2 tests
test <pkg>:<target> ... ok
test <pkg>:<target> ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s
Cabin has no ignore or benchmark mechanism, so ignored and measured are constant zeros - they
keep the summary line shaped exactly like cargo test’s. filtered out counts the test targets
in the selected packages that the invocation deselected via --test <NAME> or skipped because
their required-features are not enabled. finished in is
the wall-clock time of the test run (the build is not included).
A test executable’s stdout / stderr appear under ---- stdout: <pkg>:<target> ---- / ---- stderr: <pkg>:<target> ---- headers. With --test-jobs 1 they stream live while the test runs;
unlike cargo test, output is not buffered until the end of the run. With more test jobs, each
executable’s output is printed in one piece when it finishes, just before its result line, so
output from concurrent tests never interleaves. Result lines appear in the order tests finish.
A test that does not pass is reported with the reason:
test <pkg>:<target> ... FAILED (exit 17)
test <pkg>:<target> ... FAILED (terminated by signal 11)
test <pkg>:<target> ... FAILED (timed out after 60s)
test <pkg>:<target> ... FAILED (could not start: Permission denied (os error 13))
Signals are reported on Unix only. A result line ends in (output cut off) when Cabin stopped
reading output that a leftover process held open (see timeouts).
If any test fails, cabin test itself exits non-zero. A failures: recap lists the failed test
names before the summary, followed by the top-level error on stderr:
failures:
<pkg>:<target>
test result: FAILED. P passed; F failed; 0 ignored; 0 measured; FO filtered out; finished in T.TTs
error: test failures: F of N test executables failed
JUnit reports
--junit <PATH> writes a JUnit XML report once the tests have run, whether or not they passed; CI
services that read JUnit can show it next to the job. Cabin sees test executables, not the test
cases inside them, so each executable is one testcase (named after the target, with the package as
classname) and each package one testsuite. Each case carries the executable’s run time and its
stdout and stderr as system-out and system-err. A test that fails, is terminated by a signal,
or times out is a failure whose message is the reason the result line shows (exit 17, timed out after 60s); a test that cannot start is an error. Output is decoded as UTF-8, with replacement
characters for invalid bytes and for U+FFFE and U+FFFF, which XML cannot carry. ANSI escapes and
the C0 and C1 control characters other than the line feed are dropped, tabs and carriage returns
included, and an escape sequence that never ends hides the rest of that stream; the terminal output
keeps everything. A test whose output was cut off has the property output-cut-off set to true.
No report is written when no test runs.
For a report per test case, pass the test framework its own reporting arguments after --, such as
GoogleTest’s --gtest_output=xml:<dir>/, which names each file after its executable; a relative
path resolves against each package’s directory, and --test limits the run to the executables that
accept the argument.
Working directory and environment
Each test executable runs with its working directory set to the owning package’s manifest directory. Tests that read fixture data from paths relative to the package root therefore see the same files in CI and on a developer’s machine.
The environment the test inherits is the same as cabin test’s own, without the registry
credentials (CABIN_REGISTRY_TOKEN, ACTIONS_ID_TOKEN_REQUEST_TOKEN) and plus the deterministic
CABIN_* package-execution overlay (manifest dir / path, package name / version, profile, build
dir) documented in environment-variables.md. Cabin does not set
test-framework-specific variables; if you need a particular variable, set it in the parent process.
Standard input is empty: tests that run at the same time cannot share a terminal.
Determinism
Tests start in (package_name, target_name) ascending order, regardless of the order targets appear
in cabin.toml, and the failures: recap uses the same order. With --test-jobs 1 each test runs
to completion before the next starts.
Dev-dependencies
cabin test activates [dev-dependencies] for the selected packages as real graph edges. A test
target may therefore depend on packages declared only under [dev-dependencies]:
[dependencies]
demo = { path = "../demo" }
[dev-dependencies]
"cabin-ports/googletest" = "=1.17.0"
[target.demo_test]
type = "test"
sources = ["tests/lib_test.cc"]
deps = ["demo", "cabin-ports/googletest"]
Every dependency source form works under [dev-dependencies]: versioned registry requirements
(resolved and fetched like normal deps, and recorded in the lockfile) and path deps. The
unit-test-gtest example
links the cabin-ports/googletest registry package from [dev-dependencies] this way.
Only dev-only target kinds (test, example) may list dev dependencies in deps; an ordinary
library / executable target referencing one fails with a diagnostic naming the
[dev-dependencies] policy, so a production target cannot accidentally link test-only code.
cabin build continues to ignore [dev-dependencies], so ordinary production builds remain
unaffected. Dev-dep activation never propagates: a transitive package’s own [dev-dependencies]
stay declaration-only even under cabin test.
Lockfile behavior
--locked and --frozen apply to cabin test exactly as they do to cabin build:
--lockedrequires the lockfile to satisfy the resolver’s pick (and the patch / source-replacement state to match);--frozenadditionally forbids state-writing side effects (cache population, lockfile mutation).
Because cabin test includes [dev-dependencies] in resolution, projects that have never run
cabin test may need one un---locked invocation to add dev-deps to the lockfile.
What cabin test is not
- It does not run examples.
exampletargets are not selectable from the command line - they reach the build graph only as transitive deps of another selected target. - It does not parse GoogleTest / Catch2 / doctest output; its JUnit report has one test case per executable.
- It does not filter test cases inside an executable itself -
--test <NAME>selects wholetesttargets; arguments after--let the test framework filter its own cases.
Contributing tests to Cabin
The rules for writing Cabin’s own test suite - environment isolation via the shared cabin()
helper, tool-availability gating, host-path and driver-name conventions, and the CI portability
boundary - are contributor guidance, not part of the cabin test user surface. They live in
crates/AGENTS.md.