Benchmarks
The MerRec queries compare Algeon with CPU pipelines over the 1.27-billion-event dataset. The TPC-H and TPC-DS suites at scale factor 100 compare the CPU and GPU paths, and the server load runs drive concurrent TPC-H streams through Arrow Flight SQL.
GPU algorithms
The GPU algorithms behind cugraph_*, cuvs_* and cuml_* UDTFs run NVIDIA's native
libraries inside DataFusion physical plans alongside relational operators. NVIDIA
publishes upstream kernel benchmarks for cuGraph,
cuVS,
and cuML.
MerRec queries against CPU pipelines
Each query reads the full six-month MerRec dataset, 1,268,092,738 events and 82,614,392 items on local NVMe, with the files in the page cache. Each CPU pipeline runs the relational steps in DuckDB 1.5.6 on 32 threads and the algorithm in the named library. Both sides ran on one host: an Intel Xeon Platinum 8559C with 32 vCPUs, 256 GiB RAM and an NVIDIA RTX PRO 6000 with 96 GB.
The table lists every algorithm query in the MerRec guide. Algeon times are
medians of runs 2 to 4 over Flight SQL with a release binary and no resident
GPU cache (cache_cap_bytes = 0); CPU times are medians of three runs, none
excluded. The data file keeps the first GPU run of each query, which also
pays process start-up costs.
Stage breakdown across the eight queries
- Barbie tie-ins builds two undirected graphs from 23 million distinct
user-brand pairs and runs personalized and global PageRank on each. The CPU
pipeline takes 2.3 s in the DuckDB join, 13.0 s renumbering vertices and
building the igraph graphs, and 7.7 s in the four PageRank calls. NetworkX
3.7 takes 158 s once its tolerance matches cuGraph's absolute L1 tolerance
(
tol = 1e-5 / N); its defaulttol=1e-6is a mean per-vertex error and stops after two iterations on 2.1 million vertices. - Session audit finds weakly connected components over 39.8 million
user-session edges. DuckDB deduplication and renumbering take 9.3 s and
scipy
connected_componentstakes 3.0 s. - Audience funnel combines graph affinity, taste vectors and purchase propensity in one plan. The CPU pipeline passes intermediate tables between igraph, FAISS and XGBoost in host memory; graph construction and personalized PageRank take 14 s. Shoppers tied at the 20th-neighbor distance fall on either side of the cutoff, and the lookalike count varies between 19,096 and 19,104.
- Propensity model takes 14.7 s in DuckDB to aggregate features over the events before September, and that aggregation takes most of the 6.54 s on the GPU. Forest inference takes 1.1 s in XGBoost and under 0.01 s in cuML. Tied scores at the shortlist cutoff give 16,087 buyers on the GPU and 16,090 on the CPU.
- Brand twins compares the fan sets of the 2,000 largest brands by Jaccard similarity. The DuckDB self-join takes 2.9 s to collect the fan sets and 3.4 s to join them.
- Pokemon lookalikes builds a 64-dimensional taste vector for each of 2.1 million shoppers in SQL, then searches 20 neighbors of each seed buyer. DuckDB takes 19.5 s to build the vectors. The search takes 6.4 s in FAISS and 0.03 s in cuVS.
- Shopping worlds groups the 5,000 largest brands into worlds with Leiden over a Jaccard similarity graph. DuckDB takes 8.3 s for the fan sets and Jaccard, and igraph 0.5 s for Leiden. The two Leiden implementations return different partitions; the result lists are not compared row by row.
- World bridges scores 12.5 million brand pairs with Jaccard similarity, splits the similarity graph into worlds with Leiden and ranks bridge brands by exact betweenness. The DuckDB self-join takes 6.2 s and NetworKit's betweenness 1.7 s.
Measurement details and reproduction
The GPU server runs a release build for
x86-64-v3 on driver
595.91.07 and CUDA
13.2, configured by
server.toml.
GPU and FAISS medians exclude the first of four runs. The 100,000-query FAISS
case ran five times and also excludes its second run, which overlapped a
compile job on the same host. FAISS used 8 threads,
chosen in a tuning run against 16 and 32, with NumPy 2.5.3 and PyArrow 25.0.1. The
algorithm scale guide
documents SQL generation, CPU tuning and result validation, and the
data file
keeps every run.
Exact kNN over 2.1 million vectors
Each search finds the 10 nearest of the
2,080,74364-dimensional Float32 taste vectors
in mr_taste64 for the first 100 to 100,000 shoppers. The distance column
counts query rows times taste rows.
Algeon runs the search through the cuvs_brute_force_knn table function in
SQL. The Algeon time covers the whole query, from the Parquet scan on local
NVMe to the last result batch over Flight SQL. The CPU time is FAISS 1.15.1
IndexFlatL2 on 8 threads,
timed from Parquet load to the Arrow result.
GPU and FAISS neighbor distances agree within 1e-6 after both are recomputed in Float64. Neighbors at tied distances may differ by ID.
Item-user PageRank
The query below builds a weighted, undirected graph from six months of item
views. It scans the 1,268,092,738 events, groups the item views by
(item_id, user_id) for the interaction weights, and runs cugraph_pagerank
over the resulting 87,055,073 vertices and
1,682,503,472 directed edges.
PageRank converges in 75 iterations at the default
alpha => 0.85 and epsilon => 1e-5.
SELECT vertex, value
FROM cugraph_pagerank(
edges => (
SELECT item_id AS src, -(user_id + 1) AS dst,
CAST(COUNT(*) AS DOUBLE) AS weight
FROM merrec_events
WHERE event_type = 'item_view'
GROUP BY item_id, user_id
),
directed => false
)
WHERE vertex >= 0
ORDER BY value DESC, vertex
LIMIT 20;
The whole query, edge preparation included, takes a median of
20.17 s; the cuGraph PageRank call
accounts for 7.92 s of it. The aggregated
edges and the graph stay in device memory between the GROUP BY and the
PageRank call.
Relational path
The TPC-H and TPC-DS suites at scale factor 100, CPU path against GPU path, one warm-up and one measured iteration per query, on an AWS EC2 g7e.8xlarge. These runs are derived from the TPC-H and TPC-DS benchmarks. They are not audited TPC results and are not comparable to published TPC results. TPC-H and TPC-DS are trademarks of the Transaction Processing Performance Council. The query text is the DuckDB rendering of the TPC specifications (MIT).
- CPU: Intel Xeon Platinum 8559C, 32 vCPU.
- Memory: 256 GiB.
- GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition, 96 GB.
- Storage: local NVMe Parquet files. GPU reads use KvikIO compatibility mode with Algeon's default 64 I/O threads, without GPUDirect Storage. CPU reads use DataFusion's Arrow Parquet reader.
- DataFusion version
55.1.
Both paths read the Parquet files from the OS page cache. Nothing was flushed
between runs, and no cold-storage run was measured. The GPU keeps no data
between queries (cache_cap_bytes = 0), so each measured iteration reads the
column pages again, copies them to the device and decodes them. The TPC-DS
CPU run reuses one DataFusion session for all 99 queries, and its Parquet
footer cache can keep footers across the warm-up and the measured iteration;
the GPU path reads footers again on each plan. Caches along the
path describes what each layer holds.
TPC-H SF100
All 22 queries ran on the GPU path, and every output matched the CPU path.
- Result: CPU 47.9s, GPU 13.9s, CPU / GPU 3.5x, largest task memory peak 27.0 GiB.
- Per-query runtime
- Speedup
- Memory pressure
- Per-query results
Wall time of the single measured iteration after one warm-up. Parquet in the OS page cache; no GPU resident cache.
CPU wall time divided by GPU wall time.
Largest cuDF allocator peak that one compiled task reported, per query.
22 TPC-H queries.
TPC-DS SF100
All 99 queries ran on the GPU path in NativeRequired mode, and every result matched the CPU path.
- Result: CPU 169.6s, GPU 51.1s across 99/99 successful native queries, CPU / GPU 3.32x, largest task memory peak 38.7 GiB.
- Per-query runtime
- Speedup
- Memory pressure
- Per-query results
Wall time of the single measured iteration after one warm-up. Parquet in the OS page cache; no GPU resident cache. The Y-axis skips 15–60s.
Scroll horizontally to inspect all 99 queries.
CPU wall / successful GPU wall, sorted ascending. Green is GPU faster; red is GPU slower.
Largest cuDF allocator peak that one compiled task reported, per query.
99 TPC-DS queries. GPU faster on 69; slower on 30.
GPU execution was faster for 69 queries and CPU for 30. The largest absolute CPU lead was Q44 (0.53s CPU, 1.10s GPU). The CPU / GPU ratio sums the measured query times under the tuned CPU configuration below.
DataFusion CPU baseline configuration and tuning
The TPC-DS CPU baseline ran DataFusion 55.1.0 with these settings. Settings not listed keep their defaults. The TPC-H run above used 8,192-row batches.
Five CPU profiles were screened on Q03, Q04, Q07, Q23, Q64, Q74, Q75 and Q97, one iteration each without warm-up. The fastest became the baseline above. Each figure is one iteration on one machine.
TPC-DS reproduction and parity verification
Both paths use release builds with target-cpu=x86-64-v3 and the same 24
Parquet files, totalling 41,372,641,702 bytes and 959,037,905 rows. No caches
were flushed. Each query's warm-up precedes its measured execution. wall_ms
measures execution through result collection, including reads and GPU result
transfer. Successful-query SQL and physical planning time are excluded.
GPU runs use the sf100_gpu_throughput profile with
parquet_throughput source planning, an explicit 32 GiB
max_source_chunk_bytes target and at most 1,024 row groups per chunk.
The byte target groups source data; the query's separate
memory grant limits
device allocations. This benchmark profile sets its row-group cap directly.
The server's default chunk policy
derives its byte target from the admitted grant. Statistical aggregates use
gpu_native_tolerant.
The TPC-DS environment variables select the CPU partition and batch settings:
source env.sh
export ALGEON_TPCDS_CPU_TARGET_PARTITIONS=32
export ALGEON_TPCDS_CPU_BATCH_SIZE=32768
export ALGEON_TPCDS_NATIVE_PROFILE=sf100_gpu_throughput
export ALGEON_TPCDS_NATIVE_STATISTICAL_AGGREGATE_EXECUTION_MODE=gpu_native_tolerant
# Point this at the SF100 Parquet directory on local NVMe.
export ALGEON_PARQUET_DIR=/mnt/algeon-nvme/algeon/fixture/tpcds/sf100/parquet
bash scripts/dev/bench_one.sh tpcds all --engine cpu_datafusion \
--scale 100 --warmups 1 --iterations 1 \
--parquet-dir "$ALGEON_PARQUET_DIR" --out /tmp/tpcds-cpu.tsv
bash scripts/dev/bench_one.sh tpcds all --engine native_gpu \
--scale 100 --warmups 1 --iterations 1 \
--parquet-dir "$ALGEON_PARQUET_DIR" --out /tmp/tpcds-gpu.tsv
native_tpcds_parity_report checks the result values. It runs each query on
both paths in one process, with the CPU side on the same partition and batch
settings, and compares every value. A top-level ORDER BY preserves the
sequence of sorting keys. Rows may exchange positions only within contiguous
groups equal on every key; unresolved keys retain emitted row order. Queries
without top-level ordering use multiset comparison, preserving duplicate
counts. Tie groups require exact key equality. Floating-point
columns, which are the GPU variance and standard deviation outputs under
gpu_native_tolerant, accept an absolute and relative error of
1e-8;
every other column must match exactly. The result-values column in the table
names the mode used for each query.
export ALGEON_TPCDS_PARITY_FLOAT_ABS_TOLERANCE=0.00000001
export ALGEON_TPCDS_PARITY_FLOAT_REL_TOLERANCE=0.00000001
flock /tmp/cudf-gpu.lock cargo run --release -p bench --all-features \
--bin native_tpcds_parity_report > /tmp/tpcds-parity.tsv
The generator refuses a parity TSV whose dataset root, query set or CPU settings differ from the timing run:
python3 website/scripts/generate_tpcds_sf100_benchmark_ts.py \
--cpu-tsv /tmp/tpcds-cpu.tsv --gpu-tsv /tmp/tpcds-gpu.tsv \
--parity-tsv /tmp/tpcds-parity.tsv \
--environment /tmp/tpcds-environment.json \
--out website/src/data/tpcdsSf100Benchmark.ts
Concurrent TPC-H streams through Algeon server
The runs above time one query at a time inside a benchmark process. In this
run, 1 to 16 concurrent Flight SQL clients ran the 22 TPC-H SF100 queries
against Algeon server on the same g7e.8xlarge. Each client ran one stream: the 22
queries in its own seeded order, twice, with the first pass as warm-up. The
GPU server ran every query in native_required mode. The CPU path is the same
binary with execution_mode = "functions_only", which plans every query
through DataFusion on the CPU. Queries that call GPU UDTFs go through the same
admission queue and memory caps; this run uses relational SQL only.
- Throughput
- p95 latency
- All values
Completed queries per second over the measured pass, median of 3 runs. Parquet in the OS page cache before each run.
95th percentile query latency in seconds, from request to the last result batch.
Median of 3 runs with the range in parentheses.
With the server's default settings, the GPU peaked at 2.11 queries per second (7,596 per hour) at 16 streams. Throughput was within 2% of that peak from 4 streams up, while median latency rose from 1.7 s to 7.6 s at 16 streams. The CPU path peaked at 0.54 queries per second, and the GPU default completed 3.9 times as many.
The admission setting max_active_attempts sets where throughput levels off.
The server runs up to that many queries at once and queues later submissions
in FIFO order. Each admitted query receives a
fixed memory cap equal to device managed capacity divided by max_active_attempts:
94.4 GiB at concurrency limit 1,
47.2 GiB at the default limit 2, and
31.5 GiB at limit 3 on this GPU. Concurrency limit 1
stayed near 1.38 queries per second from
4 streams up. Concurrency limit 3 reached
2.48,
18%
above the default. All 22 queries completed under each of the three caps. A
query that allocates more than its cap fails with query_memory_limit_exceeded
while the other admitted queries continue. Raising max_active_attempts raises
throughput for queries that fit the smaller cap and lowers the largest query the
server can admit. Memory and query lifetime
describes the queue, the caps and how the allocator enforces them.
Server overload and admission queue behavior
In these overload runs, the client submitted queries to the default configuration at sustained rates from 1.0× to 2.0× of rated capacity (2.11 queries per second) for five minutes after 30 seconds of warm-up. The client submitted each query on schedule whether or not earlier queries had finished.
Completed throughput held between
2.06 and
2.07 queries per
second from 1.0× to 2.0× offered load. The server rejected the excess with
service_overloaded. With all 64 queue positions
occupied, a new query was rejected after a median of 50 ms.
A queued query that waited longer than max_queue_wait_ms (30 s)
was rejected at that point. At 1.5× load and above, admitted queries waited in the queue
before execution, and p95 completion latency reached
31.5 s.
Every admitted query result matched the CPU reference output. After each run, the server
reported no active queries and no remaining memory grants.
Server load measurement methodology and reproduction
- Client and server ran on one machine over loopback.
tasksetpinned the client to 4 vCPUs and the server to the other 28. The Parquet files were fully resident in the page cache before each run. - The server runs a release build configured by
tpch-sf100-server.toml, which sets onlynative_requiredand the table registrations. A=1 and A=3 add one[[admission.device_profiles]]entry. The CPU path ran withdatafusion.execution.target_partitions = 32anddatafusion.execution.batch_size = 32768. - Each point is the median of 3 runs, with
a fresh server for each run. Throughput counts queries completed in the
measured pass, from the moment every stream finished warm-up to the last
completion. Latency runs from
GetFlightInfoto the last result batch. - Every result was compared with references recorded by the CPU path.
19,777 GPU
results matched, warm-up passes included. Q11 returns no rows at SF100
because the DuckDB query text fixes its
FRACTIONat 0.0001 where the specification uses 0.0001 / SF; its check compares only the schema. - This run is derived from TPC-H and is not a TPC-H Throughput Test. It runs no refresh functions, orders each stream from seed 20261013 and reports no TPC metric.
- The stress runbook
documents the server, reference, streams and overload commands for
algeon_tools flight-sql-stress.