Tests

Use the targeted edit loops during development and the full gate for pre-merge validation or an explicit request; documentation-only changes need no Cargo or GPU checks. GPU checks run locally. GitHub Actions runs only the native-free checks, because hosted runners have no NVIDIA GPU. Test placement is defined in Test layout. Script maintainers use the harness reference.

Before running Cargo checks:

• The build shell is configured and all four native layers plus the Rust workspace are built.
• The gate tools from Prerequisites are installed: the pinned nightly toolchain, the latest stable cargo-nextest, cargo-hack 0.6.45, cargo-public-api, bindgen-cli, and PyYAML. The gate checks that nextest runs and prints its version. Nextest is not pinned. The full and --native-free modes also check the exact cargo-hack version, and the full mode checks cargo-public-api.
• No benchmark, running Flight SQL server, or other checkout's gate holds /tmp/cudf-gpu.lock, since GPU tests wait on that lock until it is free.

Set the native loader directories once before the Cargo commands below. The gate exports this same value for every phase, replacing any caller value:

export LD_LIBRARY_PATH="$PWD/target/native/cudf-install/lib:$PWD/target/native/cuvs-install/lib:$PWD/target/native/cugraph-install/lib:$PWD/target/native/cuml-install/lib:$PWD/target/native/cugraph-build"

Repository executables receive native rpath arguments from their own package build scripts. Installed cuGraph and cuML libraries also name their cuVS and RMM dependency prefixes in RUNPATH. The gate exports the loader directories for all phases, including doctests whose binaries carry no native rpath. A downstream application does not inherit dependency build-script rpath arguments. Its build and direct execution are covered separately by the Building for an Application and the registry consumer check.

Development checks

Targeted edit loops

Ordinary Cargo build, check, lint and test commands run from the repository root with --workspace --all-features. cargo nt is the alias for cargo nextest run --workspace --all-features defined in .cargo/config.toml. Select tests with filtersets, never -p, to reuse that build configuration:

cargo nt -E 'package(<package>)'                              # what you changed
cargo nt -E 'rdeps(<package>)'                                # everything downstream
cargo nt -E 'binary_id(<package>::<target>) & test(<name>)'   # one test
cargo nt -E 'not group(gpu) & not group(gpu-exclusive)'       # CPU only
cargo nt -E 'package(bench)'                                  # the non-default bench package
cargo nt -E '<same filterset>' -R latest                      # rerun only failures

A rerun must repeat the original filterset, because nextest treats binaries that a different or missing -E skipped earlier as new and reruns the whole suite. Plain cargo test is reserved for doctests. Because libtest executes test cases as concurrent threads within a single process, it cannot reliably restore the process-global allocator or backend policy fixtures; full validation therefore relies on cargo-nextest for per-process isolation. The test lanes live in .config/nextest.toml.

Formatting, lints, and docs match the gate's phases:

cargo +"$(cat .config/policies/nightly-toolchain.txt)" fmt --all --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
RUSTDOCFLAGS="-D warnings" cargo doc --workspace --all-features --no-deps --keep-going
bash scripts/ci/check_public_api_surface.sh check --all-features

The public API check enforces the API documentation boundary alongside the signature snapshots. Updating a snapshot does not waive missing documentation on a reachable owner type. The isolated documentation regression tests run as part of the snapshot policy self-test:

bash scripts/ci/tests/check_public_api_surface_tests.sh

Broader CPU checks

--fast runs rustfmt, the native-source check, the policy self-test scripts, and the CPU nextest tests (cargo nt -E 'not group(gpu) & not group(gpu-exclusive)'). It skips msrv-check, feature-isolation, the shipped-server clippy lane, all GPU tests, doctests, rustdoc, the public API snapshot, the hidden-API consumer probe, and clippy.

bash scripts/check_all.sh --fast

Native-free checks

--native-free runs the checks that need neither a GPU nor the native RAPIDS builds. GitHub Actions runs this mode on every pull request. Every Cargo phase runs with DOCS_RS=1, which makes the -sys build scripts skip native discovery and linking, and uses target/native-free as its target directory. The mode runs rustfmt, the native-source check, the policy self-test scripts, clippy with orphan-source verification, strict rustdoc, shipped-server clippy, msrv, docsrs-native-free, the API consumer probe and the publication policy. Its tests are those of the packages whose test builds reach no workspace member with a links key, such as algeon-query-planning. It skips the GPU tests, doctests, every other CPU test, checked bindings, feature-isolation and the public API snapshots. The native-source check needs all four submodules checked out.

bash scripts/check_all.sh --native-free

The harness defines its workspace checks and coverage exceptions.

The full gate

Run it from the checkout root. Do not wrap nextest or the full gate in flock. The run-wrapper owns test locking. Non-test GPU commands, including benchmarks, servers and tools, take the lock separately.

bash scripts/check_all.sh

For a run that outlives a terminal or tool call, launch it with nohup and poll target/check-all/latest/summary.tsv. With no arguments the script starts policy checks and the test-build phase, which captures Cargo metadata and builds the test binaries once. The other groups start after that phase completes:

Group	What runs	Order
policy	rustfmt, native-source verification, checked bindings, and the policy self-test scripts	Independent, in parallel
tests	cargo nextest run -P gate reusing that run's Cargo and binaries metadata, then doctests through the GPU lock wrapper	Doctests start after nextest finishes. A failed test build skips nextest execution and still runs doctests
static	clippy with -D warnings, current-run orphan-source verification, strict rustdoc, then the public API snapshots and application documentation check	In sequence. Cargo and API checks use nice -n 10
hygiene	seven feature-isolation shards and their union check, shipped-server clippy, msrv, the DOCS_RS=1 workspace compile check, docsrs-native-free rustdoc over the packaged publication closure, the API consumer probe and publication policy	Independent after the test build. Compilation uses nice -n 10 and separate target directories

• A failed phase in any group fails the gate.
• Nextest reuses the test build's Cargo and binaries metadata, which fix the same full-workspace, all-features graph as cargo nt.
• Workspace Cargo phases normally use --workspace --all-features. Feature-isolation and shipped-server clippy use their defined feature selections (Feature coverage). docsrs-native-free extracts the release archives and documents each crate separately with its own docs.rs metadata features, an isolated native-free environment and strict warnings. The packaged release checks define that archive lane. The feature-off clippy supplement is described under Layout checks.

The full pre-merge gate includes packaged native-free rustdoc. The native archive consumer gate runs separately:

bash scripts/ci/check_registry_consumer.sh

It requires the production native installs and a GPU. Its complete matrix builds external applications against extracted release archives, checks loader resolution and directly executes result assertions from another working directory. See the packaged release checks for native/toolchain selectors, evidence output and the fresh-resolution and source-lockfile lanes. A focused --features or --lane rerun covers that selection; the default complete matrix establishes the archive consumer contract. Registry downloads without patches and hosted docs.rs results remain post-publication checks.

Passing run

Each phase logs to target/check-all/run.*/<phase>.log, and target/check-all/latest symlinks the newest run. Its summary.tsv has the columns phase, status, started, duration and log, with gate as the first row, and the script rewrites it every time a phase starts or finishes. A passing full run ends with:

OK: full pre-merge gate passed.

Only a complete gate run on the final source state is full validation; --fast ends with OK: fast checks passed. PARTIAL: this is not full pre-merge validation.

On failure

The run continues past a failed phase, prints each phase's status and duration, then lists the failed phases with their log paths. A failed tests or cpu-tests phase also prints its nextest summary and the rerun command: cargo nt -P gate -R latest for tests, or cargo nt -E 'not group(gpu) & not group(gpu-exclusive)' -R latest for cpu-tests. Either reruns only what that run left failed or unfinished. The script exits non-zero after:

FAIL: full pre-merge gate reported problems.

• A native-library or CUDA-architecture failure in the tests phase usually means a stale or mismatched prefix.
• A single failing phase is fastest to iterate on with the targeted edit loops. Rerun the full gate when completing pre-merge validation.
• Exit status 2 before any phase means an argument or environment error, a missing tool, or a tool-version mismatch, and the message names the fix.
• cargo nextest replay replays a stored run's results for each test without rerunning anything.

Doctests

Doctests run as a separate command through the GPU lock wrapper, since rustdoc does not build through nextest. rustdoc does not pass the build scripts' rpath link arguments to doctest binaries, and runs them from temporary directories. The loader therefore needs absolute native library directories.

Doctest command

NEXTEST_TEST_GROUP=gpu scripts/ci/nextest_gpu_lock.sh /tmp/cudf-gpu.lock \
  cargo test --workspace --all-features --doc --no-fail-fast

Multi-GPU tests

Multi-GPU cases are ignored by default and require at least two visible devices. On a host with that inventory, select them explicitly:

cargo nt --run-ignored only -E '
binary(/multi_gpu|multi_device/)
  | (binary_id(algeon-cudf::interop_contracts_gpu) & test(/::(device_event_export_rejects_readiness_from_another_device|helper_built_exports_reject_cross_device_inputs)$/))
  | (binary_id(algeon-cudf) & test(cross_device_secondary_is_rejected_with_public_classification))
  | (binary_id(algeon-query-runtime) & test(/^runtime::cache::local_parquet::tests::source_copy_rejects_a_cache_domain_on_another_device$/))
  | (binary_id(algeon-query-runtime) & test(/(multi_device_registry::|device_local_cleanup_fault_seals_retry_on_a_frozen_eligible_alternate|queued_placement_folds_a_native_fault_into_the_ledger_before_retargeting|query_admission_waiter_timeout_reports_the_selected_device_requirement)/))
'

These cases assert the device precondition when explicitly run. A single-GPU invocation reports a precondition failure.

Feature flags

Feature	Enables
cugraph	cuGraph graph algorithms callable from SQL
cuvs	cuVS UDTFs (exact kNN, KMeans, PCA) and their GPU execution
cuml	cuML UDTFs (truncated SVD, HDBSCAN, UMAP, logistic regression, random forests) and their GPU execution, which requires the native cuML install from scripts/build/build_libcuml.sh
nvml	Optional NVML device diagnostics; see NVML diagnostics

The Arrow Flight SQL server is launched by the separate algeon-server package. Its cugraph, cuvs, cuml, and nvml features forward the corresponding adapter and engine capabilities. bench and tools also declare nvml. Their run features do not enable it, and tools/nvml forwards only to a backend that another selected feature already includes:

cargo build -p bench --features nvml
cargo build -p tools --features single-node-multi-gpu-validate-run,nvml

Cargo merges the features of one invocation. --workspace --all-features therefore always includes NVML. The feature-isolation phase runs scripts/ci/check_nvml_opt_in.py, which resolves each package selection separately and rejects NVML outside an nvml selection.

cargo run -p algeon-server --all-features --bin algeon_server

Acceptance runs outside the gate

Runs that need a real dataset, cloud account, running catalog, or Python cuGraph stay outside Cargo test targets, and each one takes the GPU lock itself.

Purpose	Command	Requires
TPC-DS parity report	ALGEON_PARQUET_DIR=/path/to/complete/tpcds-sf1 flock /tmp/cudf-gpu.lock cargo run -p bench --all-features --bin native_tpcds_parity_report	A complete TPC-DS dataset. ALGEON_TPCDS_CPU_TARGET_PARTITIONS (default 16) and ALGEON_TPCDS_CPU_BATCH_SIZE (default 8192) configure the CPU oracle, with the same meaning as in tpcds_gpu_cpu_bench
TPC-H execution report	ALGEON_PARQUET_DIR=/path/to/complete/tpch-sf1 flock /tmp/cudf-gpu.lock cargo run -p bench --all-features --bin native_tpch_execution_report	A complete TPC-H dataset
Sustained-load goodput (query_service_goodput)	crates/bench/scripts/run_goodput_matrix.sh calibrate g1	The GPU and dataset recorded by calibrate
Python-cuGraph comparison	flock /tmp/cudf-gpu.lock cargo run -p tools --all-features -- cugraph-fixture-e2e bfs-include-edges	A host with Python cuGraph installed
S3 Parquet scans	ALGEON_S3_PARQUET_E2E_BUCKET=<bucket> flock /tmp/cudf-gpu.lock cargo run -p tools --features s3-parquet-e2e -- s3-parquet-e2e	An existing bucket and credentials from the default AWS chain. ALGEON_S3_PARQUET_E2E_REGION defaults to AWS_REGION, then us-east-1
Iceberg REST and Glue catalogs	AWS_ACCESS_KEY_ID=algeon-e2e AWS_SECRET_ACCESS_KEY=algeon-e2e-secret flock /tmp/cudf-gpu.lock cargo run -p tools --features iceberg-local-e2e -- iceberg-local-e2e	The services of crates/tools/fixtures/iceberg-local/compose.yaml

• Scale-factor workloads validate their dataset before starting and fail when it is incomplete. The TPC-DS report emits one row per attempted query.
• The goodput driver script pins every factor except the one under comparison and records source, GPU, and native-library identity next to the results. calibrate freezes each item's single-flight median latency and output fingerprint. The measurement groups refuse to run if the live GPU or dataset differs from that calibration.
• An unavailable Python cuGraph oracle fails the run. The run does not skip it. Replace bfs-include-edges with pagerank, bfs-utf8-include-edges, bfs-utf8-path, or bfs-utf8-predicates for the other provisioned cases.
• External-service E2E tests are invoked explicitly through cargo run -p tools --all-features.
• s3-parquet-e2e uploads generated Parquet objects under a unique key prefix, scans one object, a prefix of three objects and one object split into byte ranges in library and server-configured sessions, compares each result with the generated data, and deletes the objects. Setting ALGEON_S3_PARQUET_E2E_ENDPOINT selects an S3-compatible service with path-style addressing; without it the run uses real S3.
• iceberg-local-e2e writes Iceberg tables through a REST catalog and a Glue catalog, serves both from an in-process Flight SQL server in native_required mode, and checks the Flight SQL catalog, schema and table listings, a native aggregate over each catalog and the unsupported_table_feature rejection of a snapshot with a position-delete file. Its compose file adds the Iceberg REST fixture on http://127.0.0.1:8181 and moto as Glue on http://127.0.0.1:5001 to the RustFS service below. Both compose files bind port 9000; stop one stack before starting the other.

A local RustFS service for that run starts from the compose file in crates/tools/fixtures/s3-local. It listens on http://127.0.0.1:9000 with bucket algeon-e2e, access key algeon-e2e and secret key algeon-e2e-secret.

docker compose -f crates/tools/fixtures/s3-local/compose.yaml up -d --wait
ALGEON_S3_PARQUET_E2E_ENDPOINT=http://127.0.0.1:9000 ALGEON_S3_PARQUET_E2E_BUCKET=algeon-e2e \
AWS_ACCESS_KEY_ID=algeon-e2e AWS_SECRET_ACCESS_KEY=algeon-e2e-secret \
  flock /tmp/cudf-gpu.lock cargo run -p tools --features s3-parquet-e2e -- s3-parquet-e2e
docker compose -f crates/tools/fixtures/s3-local/compose.yaml down -v

For any performance claim, use --release builds, compare rows produced by the same harness, and label warmup count, iteration count, source cache policy, and source chunk profile. See the benchmark reference.