Handling Errors

Every Algeon-owned adapter failure carries a stable machine-readable code, a coarse kind, and a status that says whether the condition is permanent, temporary or persistent, beside a human-readable message. An application finds that identity inside a DataFusionError or in Flight SQL gRPC metadata, and should decide from those typed fields, never from formatted error text. Ordinary DataFusion errors and transport errors below the server keep their own policies. Error ownership covers how errors are created and what each surface discloses.

cuDF / cuGraph / cuVS / cuML
  -> shared native QueryError taxonomy
  -> algeon_datafusion::error::Error
  -> DataFusionError::External
  -> embedded application or Flight SQL status

One identity, several projections

Field	Use
code	Most specific stable machine identity.
kind	Coarse grouping derived from code, for protocol and aggregate policy.
status	Coarse source condition derived from code: Permanent, Temporary or Persistent.
operation	Stable dotted identifier for the operation that added this boundary context.
message	Human diagnostic text.
facts	Structured diagnostic context, raw by default.
source	Original typed cause chain.

A failure that a RAPIDS library reported, or that broke a library's input contract, has a rapids_* code such as rapids_invalid_input and a rapids_library fact with the value cudf, cugraph, cuvs or cuml. The fact appears in the server log's error_facts and in GetQueryStats as fact.rapids_library, never in Flight metadata. Retry and protocol decisions read kind and status, which do not depend on the library.

Do not parse Display or Debug output, since adapter Error::Display emits only the human message and native QueryError formatting can contain raw facts and source text for trusted in-process use.

Find the Algeon error

From<Error> for DataFusionError wraps the error once as DataFusionError::External. DataFusion may add context, diagnostics, shared errors or collections around it, and find_algeon_error() searches them:

use datafusion::common::DataFusionError;
use algeon_datafusion::error::DataFusionErrorExt;

fn record_failure(error: &DataFusionError) {
    if let Some(algeon) = error.find_algeon_error() {
        tracing::warn!(
            code = algeon.code().as_str(),
            kind = algeon.kind().as_str(),
            status = algeon.status().as_str(),
            operation = algeon.operation(),
            "Algeon query failed"
        );
        return;
    }

    tracing::warn!(error = %error, "DataFusion query failed");
}

The search walks source chains and each DataFusionError::Collection branch in deterministic depth-first pre-order, bounded by depth and total node count, and returns the first Algeon error it reaches, whatever its severity. A None result means only that none was found within that budget. In that case, maintain the application's existing DataFusion error policy.

Read the native chain

Error::trusted_diagnostic() prints the whole error on one line: code, kind, status, operation, message, the caller schema detail, every raw fact and the source chain. The line can hold query literals, object paths and column names, so print it in tests, debuggers and trusted local tools, never in operational logs or client responses.

The source of an error converted from the native engine is a QueryError. Downcast it through the adapter's native re-export, which names the exact type the adapter linked. A direct dependency on another algeon-query-planning version would never match. Typed failure evidence, such as the allocator snapshot of a memory failure, is on context():

use algeon_datafusion::error::{DataFusionErrorExt, native::QueryError};
use datafusion::common::DataFusionError;

fn inspect(error: &DataFusionError) {
    let Some(algeon) = error.find_algeon_error() else { return };
    eprintln!("{}", algeon.trusted_diagnostic());

    let native = algeon.source_error().and_then(|source| source.downcast_ref::<QueryError>());
    if let Some(usage) = native.and_then(|native| native.context().memory.as_ref()?.usage) {
        eprintln!("peak {} of {} bytes", usage.peak_bytes, usage.limit_bytes);
    }
}

The chain stops at the QueryError for Parquet and object-store reads. Their reader text can name a sensitive location, so the read boundary keeps cuDF's classification and a URI-redacted cudf_message fact but attaches no source.

Retry and recovery are evidence-based

ErrorStatus describes the source condition and grants no permission to retry:

Status	Meaning
Temporary	The condition may clear, but the error alone does not prove when or how to retry.
Persistent	A dependency failure is known or observed to continue. Stop automatic retry and escalate.
Permanent	Do not retry the same request automatically, since it normally needs different input, configuration, capability or code. Cancellation is also terminal.

There is no adapter Error::is_retryable(). To recover from an attempted query:

1. Classify the returned error with QueryAdmissionOutcome::from_error (or from_error_with_queue_bounds) while admission is still being attempted, and with from_execution_error after a handle has been published.
2. Retain the attempt completion the service published for the attempt.
3. Wait for the sealed QueryAttemptReport, read its retry_advice, and satisfy the advice's paired prerequisite before submitting a new attempt.

An execution error does not prove that cleanup finished; only the sealed report binds the advice to completed cleanup. Retry evidence lists what the advice can say.

Flight SQL status

The built-in server looks for a typed adapter error first. An ordinary DataFusion error gets a server-private classification for its gRPC code and no public Algeon identity.

Error kind	gRPC code	Client message
cancelled	CANCELLED	The error message.
not_found, permission_denied, unauthenticated	NOT_FOUND, PERMISSION_DENIED, UNAUTHENTICATED	The error message, or fixed text for a source failure.
planning, invalid_input	INVALID_ARGUMENT	The error message.
unsupported	UNIMPLEMENTED	The error message.
resource_exhausted	RESOURCE_EXHAUSTED	Text naming the change the request needs.
execution, dependency_failure, invariant_violation, internal	INTERNAL	Static text pointing at GetQueryStats.

• invalid_configuration has kind invalid_input. algorithm_did_not_converge returns FAILED_PRECONDITION.
• A query that names an unknown table or column fails in DataFusion's planner, so its code is datafusion_planning or field_not_found and the message is DataFusion's text. DROP of a missing object and an unknown GPU function are object_not_found. A CREATE EXTERNAL TABLE whose location has no object, neither a file nor a non-empty prefix, is source_object_not_found.
• The static text refers to GetQueryStats for the query id in the status's x-algeon-query-id metadata. Every failure of a registered statement (GetFlightInfo planning, DoGet, DoPut updates, prepared statement actions) carries it; metadata RPCs such as GetCatalogs do not.

Every status built from an adapter Error carries x-algeon-error-code, x-algeon-error-kind, x-algeon-error-status and x-algeon-error-operation, followed by the projected facts that apply.

The ADBC Flight SQL driver picks the Python exception class from the gRPC code alone, and it maps RESOURCE_EXHAUSTED to InternalError. A memory-cap failure therefore arrives as InternalError even though the server classified it as resource_exhausted. Read x-algeon-error-code and x-algeon-error-kind from the exception's details list of (key, value) pairs; do not branch on the exception class.

Memory failures

For query_memory_limit_exceeded, device_out_of_memory and device_allocation_failed, the message names the operation and the requested, peak and grant bytes it carries, each also in GiB. The remedy is to filter or project earlier, split the query, or raise admission.device_profiles.default_query_limit_bytes for the device, since an unchanged resubmission fails again. A device allocation failure also points at the sealed attempt report.

Fact	Meaning
native_operation, budget_operation	Code identifier of the failing operation.
requested_allocation_bytes	The allocation that failed.
budget_peak_bytes	The highest byte count recorded for the failing scope, including the failing operation's working set.
budget_bytes	The query grant.
budget_current_bytes	Usage sampled after the failing operation released its working set, kept out of the message.

Column element limit

One cuDF column holds at most 2,147,483,647 elements at each level: its rows and the child values of each list. A size_overflow caused by that limit names the column, its element count and the limit. When one relation or intermediate result would exceed it, the message also gives the row count that fits at the column's width and asks for a filter or a split into row ranges combined with UNION ALL. When a single Parquet row group exceeds it, the message gives the row group size to rewrite the file with. The column name appears only in the client message, never in the server log.

Non-converged algorithms

An algorithm_did_not_converge message names the algorithm, the iterations reached and the configured limits, with matching algorithm, iterations, max_iterations and epsilon metadata. Resubmitting the identical query reproduces the convergence failure. The message ends with the native layer's remedy, which the error carries once, as a fact. For Katz centrality the remedy is a lower alpha (below 1/lambda_max, the reciprocal of the largest adjacency eigenvalue) or a higher max_iterations; for HITS and eigenvector centrality it is a higher max_iterations or epsilon.

Strict unsupported rejection

In native_required mode, an unsupported column cannot stay on the DataFusion plan. The code is final_plan_requirement_unsatisfied, Flight returns InvalidArgument, and the message names the first blocking field in schema order, its Arrow type and role:

native_required mode rejected a plan that needs DataFusion CPU: GPU execution does not support projected column `countDetail` with Arrow type `List(Decimal128(10, 2))`

Metadata key	Example value
x-algeon-reason	unsupported_type
x-algeon-unsupported-column	countDetail
x-algeon-unsupported-column-type	List(Decimal128(10, 2))
x-algeon-unsupported-column-role	projected

GetQueryStats retains each as fact.<name>, such as fact.unsupported_column. When the first blocker is some other plan element, the message names it, or stays generic when the producing site cannot name one:

native_required mode rejected a plan that needs DataFusion CPU: GPU execution does not support cast from Int64 to Int32 in `CAST(buys AS INT)`; EXPLAIN GPU lists every blocking candidate

That status carries x-algeon-blocking-kind, from a fixed vocabulary such as cast, aggregate_function, operator or source (column_type for a column), and x-algeon-blocking-subject, the text before the instruction. Each surface carries these names under its own disclosure rules.

A rejected Parquet scan carries kind source. Its x-algeon-reason is the capability reason, for example native_source_remote_footer_unavailable when planning found no cached footer for an s3:// object, and the subject reads like Parquet scan of 2 objects in `s3://bucket/` . The reason's remedy names the planning path and settings that fill the footer cache.

Unsupported Iceberg table features

A snapshot the Iceberg scan cannot read correctly fails with unsupported_table_feature, which has kind unsupported, status permanent and Flight code UNIMPLEMENTED. The check runs in the table provider before native lowering, so every execution mode raises it.

The status carries the reason as x-algeon-reason. Its values are iceberg_delete_files, iceberg_field_id_mismatch, iceberg_missing_field_ids, iceberg_partition_column_absent, iceberg_required_column_absent, iceberg_encrypted_file, iceberg_non_parquet_file, iceberg_multiple_buckets, iceberg_unsupported_location_scheme, iceberg_storage_properties_mismatch and iceberg_column_type_mismatch. The Iceberg support table gives each trigger. The table, snapshot id, data file path and column are raw facts and stay out of metadata and logs. A retry fails the same way until the table or the [s3] settings change.

Rejections in EXPLAIN GPU

EXPLAIN GPU returns a rejected summary row where execution returns a status. Its reason_code is the x-algeon-error-code value, and detail_json.detail is the status message without the trailing EXPLAIN GPU pointer or any x-algeon-* metadata. The rejected summary row reference has the remaining rules.

Server log

The built-in server writes one failure event per failed Flight SQL statement, converted adapter error to Flight SQL status, followed by the statement's terminal line, failed Flight SQL statement. Both lines carry the statement's query_id; the failure event starts with its query span, which also carries the correlation_id when the client set one:

INFO algeon_datafusion.flight_sql.query{query_id="d4b0794e-1560-43f7-9ff8-709894078bd5" correlation_id="d06-required-list-decimal" operation="do_get_statement"}: algeon_datafusion::server::error: converted adapter error to Flight SQL status error_code="final_plan_requirement_unsatisfied" error_kind="planning" error_status="permanent" error_operation="algeon_datafusion.planner.final_plan_requirement" error_message="GPU execution does not support a projected column with an unsupported Arrow type" client_message="native_required mode rejected a plan that needs DataFusion CPU: GPU execution does not support a projected column with an unsupported Arrow type" error_facts="blocking_kind=column_type, reason=unsupported_type"
INFO algeon_datafusion.flight_sql.query{query_id="d4b0794e-1560-43f7-9ff8-709894078bd5" correlation_id="d06-required-list-decimal" operation="do_get_statement"}: algeon_datafusion::server::flight::status: failed Flight SQL statement query_id="d4b0794e-1560-43f7-9ff8-709894078bd5" outcome="failed" stage="execute_stream" grpc_code=InvalidArgument error_code="final_plan_requirement_unsatisfied" reason="unsupported_type" elapsed_ms=5

Field	Content
error_code, error_kind, error_status, error_operation	The identity the status metadata carries.
error_message	The adapter message, without caller column names or types. A failure DataFusion classified gets a fixed sentence for its code, because DataFusion's text can echo SQL tokens and list a relation's fields.
client_message	The status text when it differs from error_message, still without caller schema names.
error_cause	The deepest cause when it adds text the message lacks, such as libcugraph's or the object store's reason. DataFusion, schema, SQL parser and Arrow errors contribute no cause.
error_source_path	The input path of a read failure.
error_facts	The log-safe facts as key=value pairs: native_* classification, rapids_library, reason, blocking_kind and memory byte counts.
error_facts_truncated, error_diagnostics_truncated	Present as true when a bound shortened the facts or the cause.

A failure with status persistent, such as device_fault, logs at ERROR, because that status asks for escalation. Otherwise the level follows the error kind:

Level	Kinds
INFO	cancelled, not_found, permission_denied, unauthenticated, planning, invalid_input, unsupported
WARN	resource_exhausted, dependency_failure
ERROR	execution, invariant_violation, internal

The terminal line uses the same level. algorithm_did_not_converge has kind dependency_failure and status permanent, and logs at WARN.

The log omits raw facts, caller schema detail and statement text. Object-store URIs in messages, causes and paths are written as <redacted-uri>, and text that looks like a credential as <redacted-credential>. Local paths stay.

Call GetQueryStats with the query_id from either line. Its [structured_error] section repeats the code and the log-safe facts as fact.<name> and adds the caller-visible schema detail, the statement text with OPTIONS values masked and the capped source chain. It is the place to look when a line reports error_facts_truncated=true.

--log takes a tracing filter. --log info,algeon_datafusion=debug adds the planning report of each statement as a DEBUG line just before its failure event.

Query stats

GetQueryStats returns the retained diagnostics of one query as UTF-8 text. Its action body is a plain string. The server first reads it as a query id. When no query has that id, it reads it as a correlation id and picks the most recently registered query with that id among the caller's own queries. The response metadata carries the resolved x-algeon-query-id, and x-algeon-correlation-id when the query has one.

A client that sets its own correlation id does not need to read any response header. With the Python ADBC driver, set the header once for the connection:

from adbc_driver_flightsql import dbapi

conn = dbapi.connect(
    uri="grpc://127.0.0.1:50051",
    db_kwargs={"adbc.flight.sql.rpc.call_header.x-algeon-correlation-id": "agent-run-1"},
)

To change the id on an open connection, pass the same key to conn.adbc_connection.set_options(...) twice: first with an empty string, which removes the header, then with the new id. The driver appends a second header value otherwise, and the server rejects a request whose correlation id key has more than one value with INVALID_ARGUMENT.

GetQueryStatsSummary takes the same body and returns only the # header lines, the attempt labels and the trailer, without the planning and execution records:

import pyarrow.flight as flight

client = flight.FlightClient("grpc://127.0.0.1:50051")
results = client.do_action(flight.Action("GetQueryStatsSummary", b"agent-run-1"))
print(b"".join(result.body.to_pybytes() for result in results).decode())

# algeon_datafusion_query_stats
# query_id=d09b7593-af81-4e40-a77a-39220a119fb8
# correlation_id=agent-run-1
# outcome=succeeded
# sealed=true
# elapsed_ms=1106
# sql=SELECT ...
# attempt_id=attempt-000000000000000886c3b4a7b7f0ef0a
# attempt_disposition=native
# attempt_outcome=succeeded
...
# incomplete=false

The summary indicates whether the query completed, which device executed it, and whether retry advice applies. Call GetQueryStats when the answer needs the per-operator [planning] and [execution] rows, the [structured_error] facts or the explained plans. A simple aggregate already returns a payload of more than 100 KB.