Skip to main content

Compose algorithms with SQL.

Run graph, vector, and machine-learning algorithms in one SQL query. Join their results, filter them, and keep querying.

See a SQL exampleBuild & run

Algeon is a query engine built on Apache DataFusion. Currently accelerated on NVIDIA GPUs, it exposes NVIDIA's cuGraph, cuVS and cuML libraries as SQL table functions and runs supported relational operations on cuDF. Analysts and AI agents can change algorithm inputs and combine results in SQL as new questions emerge. Use it as a Rust library in a DataFusion session or connect through Arrow Flight SQL. It runs beside the query engines already reading the lakehouse and answers the questions they have no operator for. The design guide follows a query from SQL to GPU columns and back, and Introducing Algeon describes the motivation and application scenarios.

Compose algorithms​

Each GPU algorithm is a table function (UDTF) whose input is a subquery. The query that uses the result also builds the graph or vectors, so nothing has to be loaded into a separate graph or vector database first.

ONE SQL QUERY · ONE PHYSICAL PLANsubqueryrelationSQLWorking setfilter · join · aggregaterows selected for the analysisGPU UDTFAlgorithmgraph · vector · MLcugraph_* · cuvs_* · cuml_*ROWSResultjoin back · filter · aggregateor the next UDTF's inputONE SQL QUERY · ONE PHYSICAL PLANsubqueryrelationSQLWorking setfilter · join · aggregaterows selected for the analysisGPU UDTFAlgorithmgraph · vector · MLcugraph_* · cuvs_* · cuml_*ROWSResultjoin back · filter · aggregateor the next UDTF's input
A UDTF takes any relation that has the columns and types it requires.

The MerRec demo queries 1.27 billion Mercari marketplace events. Its campaign list comes from one SQL query that composes graph, vector and machine-learning UDTFs:

  • cugraph_personalized_pagerank scores shoppers by affinity to the Pokemon brand on the like graph.
  • cuvs_brute_force_knn finds taste lookalikes of earlier Pokemon buyers.
  • cuml_forest_predict scores both lists with a pretrained XGBoost model.
  • GROUP BY counts who in each list later bought Pokemon.

The other demo datasets apply the same pattern to a citation graph, a product catalog and a malware benchmark.

Where Algeon fits​

  • Ad-hoc graph, vector and ML pipelines on lakehouse data. One SQL query selects the input, runs the algorithm and joins the result back. A new pipeline is a new query, with no export scripts and no separate graph or vector database. Examples.
  • Analysts and AI agents testing hypotheses. A question no dashboard tracks needs many follow-up checks with other seeds, filters or controls. The MerRec case study works through one.
  • Backends for agents and data services. The Flight SQL server queues concurrent queries, lists its functions in the catalog and returns structured errors a program can act on.
  • Join- and aggregation-heavy SQL over Parquet. Supported plan fragments run on cuDF. See the benchmarks.
  • Engines built on DataFusion. Algeon adds GPU execution and the algorithm functions to an existing DataFusion session; operators it does not support stay on the CPU.

Not a fit yet: working sets larger than one GPU's memory, sources other than Parquet, and long-lived vector indexes or models. See scope and non-goals.

Algorithm families​

Three UDTF families wrap NVIDIA's GPU libraries:

CapabilitySQL interfaceCurrent use
Graph analysiscugraph_*Graph traversal, centrality, community detection, components, similarity, core structure, and ForceAtlas2 layout.
Vector analysiscuvs_*Exact and approximate kNN, pairwise distances, per-row top-k, KMeans, and PCA over vector columns.
Machine learningcuml_*Fit models within a query, or score features with a pretrained XGBoost, LightGBM, or Treelite forest.

Build and run​

Algeon supports Linux with an NVIDIA GPU. The published server image runs the Flight SQL server without a source build. Follow Build & Test for prerequisites and build instructions, or Run as a Flight SQL server to build only the server. Then choose an integration:

Before adding a UDTF to a query, discover and validate it against registered tables. When an attempt fails, structured errors carry diagnostic fields and retry guidance.

DataFusion integration​

unselected candidatesstay on DataFusion CPUGPU resultsKvikIOYOUR QUERY INTERFACESQL · DataFrame · Flight SQLembedded in a Rust service · remote ADBC clientsALGEON OPTIMIZER RULE IN DATAFUSIONAlgeon DataFusion adapterpicks whole relational candidates for the GPUSAME PROCESS · ONE MEMORY GRANTGPU executioncuDF for the relational plancugraph_* · cuvs_* · cuml_* UDTFsSAME CALLER CONTRACTArrow batchesrows back to the interface you started fromLOCAL FILESLocal diskParquet · read-onlyS3-COMPATIBLEObject storage(WIP)Parquet · read-only
A query enters through a DataFusion session or the Flight SQL server. Algeon's DataFusion optimizer rule replaces supported plan candidates with GPU execution in the same process, and results return as ordinary Arrow batches. Table data comes from Parquet files on local disk or in S3-compatible object storage.

The execution mode sets how much relational SQL joins the GPU. By default, supported candidates execute on cuDF, while unsupported operators remain on DataFusion's CPU execution path; functions_only keeps ordinary SQL on the CPU around explicit GPU UDTFs, and native_required rejects a plan that still contains CPU operators. Explicit GPU UDTFs have no CPU fallback, and once GPU execution starts, failures return as structured errors and are never replayed on the CPU.

Experimental

APIs and query behavior may change incompatibly. Stability and result verification remain incomplete. Compare outputs against trusted reference results. The roadmap lists what changes next.

Scope and non-goals​

Algeon is a single-process engine for exploratory analysis. Today each query runs on one GPU, and the limits below follow from that.

  • Each query runs inside a fixed memory grant on its GPU, with no spilling. Graph construction and algorithm scratch space count toward the grant.
  • GPU scans read Parquet from local disk or S3-compatible storage, read-only, and cover a subset of SQL and data types. Iceberg tables from REST and Glue catalogs read as Parquet scans; see Iceberg support.
  • Indexes, fitted models and graphs exist only while the query that builds them runs. A bounded server can cache graphs.
  • DataFusion parses SQL, builds and optimizes the logical plan, and produces the physical plan. Algeon's physical optimizer rule lowers supported fragments of that completed plan to a GPU plan and optimizes the GPU plan. It has no logical planning or optimization of its own.
  • Relational operators come from cuDF and the algorithms from cuGraph, cuVS and cuML. Algeon writes no GPU kernels beyond a few element-wise routines in its cuDF bindings.
  • The design target is an agent or analyst running many ad-hoc OLAP queries against a shared GPU and revising most of them. The engine owns the physical plan, failures return structured errors with retry guidance, and admission protects the device. Sub-millisecond latency optimization falls outside this design scope. See the design priorities.