Memory and query lifetime

Algeon separates admission from execution: the query runtime commits an immutable byte grant, and the query allocator enforces that cap against device allocations.

Admission

The first poll waits for admission on the caller's task, then runs the whole GPU schedule as one pool job. Each later poll submits one job that moves at most 4 MiB to the host. Release follows the terminal outcome and the cleanup thread's synchronize.

An attempt is one execution lifecycle. Its first native work claims one compatible device and one immutable grant. A process-scoped backend shares admission across sessions, including embedded and server use. The bounded queue preserves request order. A waiting attempt holds a task on the caller's runtime until a grant is placed, the attempt is cancelled or the queue deadline passes. Missing device capabilities produce a structured error.

Below, A_d is the device's max_active_attempts and S_d its source_work_per_attempt.

Step	Runs on	Waits on	Bounded by
Plan and admission	The task polling the result stream on the host runtime.	The admission queue.	max_queued_attempts waiters and max_queue_wait_ms.
Envelope	A ContextSetup job on the GPU execution pool.	A lifecycle job permit. Cancellation ends the wait.	A_d lifecycle job permits.
Execute the schedule	A QueryExecution job on one pool worker, plus blocking-lane jobs for chunk work.	Joins on lane jobs and source reads.	A_d lifecycle permits, and S_d auxiliary permits for lanes and reads.
Each result chunk	One pool job per poll (result egress).	cudaStreamSynchronize for the device-to-host copy.	4 MiB per chunk and one MemoryPool reservation per stream.
Terminal and release	A Cleanup job on the pool, then the stream cleanup thread.	cudaStreamSynchronize on each leased stream.	A_d release cleanup permits.

The first poll of a fragment's DataFusion stream obtains the attempt's grant and runs the fragment's whole GPU schedule. The result stays on the device. Each later poll submits one job that moves at most 4 MiB to host memory. The device result and the grant are held until the stream reaches EOF, errors or is dropped.

Every production query-engine execution belongs to an attempt. Plain DataFusion execution creates an internal attempt at its execution boundary.

An attempt owns one admission claim. All its native fragments share the resulting grant, including sequential execution of a CTAS input and its result. Deferred refinement stays within that grant. A fragment planned after the claim joins when the granted device provides all its required capabilities. Otherwise it returns a typed error. Each fragment runs one compiled schedule.

A second claim returns AttemptLifecycleError::AdmissionAlreadyClaimed before any queue or ledger mutation. The existing grant and attempt lifecycle remain unchanged. The admission claim contract and the release tests cover this boundary.

Admission uses operator capability requirements. query-planning proves executability; query-runtime admits the attempt against the selected device's ledger.

An attempt executes entirely on its assigned GPU; ledger capacity and grants do not span devices.

Capacity and allocation

Each device has one authoritative ledger for committed grants, cached resident data and active-attempt slots. Its integer accounting does not infer which allocations are resident. Admission commits the service-configured cap without predicting the query's peak. For example, with 16 GiB available for grants and an 8 GiB query cap, memory accounting permits two grants, even when their current allocations total only 3 GiB. Other service limits can reduce that concurrency.

With no explicit default_query_limit_bytes, the installed profile derives the default cap as floor((M - K) / A_d). M is managed device capacity after reserved memory, K is the configured cache share, and A_d is max_active_attempts. Admission can adjust that cap for frozen resident data; an explicitly configured default is validated against the available capacity. The admission settings define those capacity controls.

For example, suppose M - K is 48 GiB and there is no frozen resident data or explicit query cap. A profile with A_d = 1 derives a 48 GiB default grant; A_d = 2 derives 24 GiB. The latter remains 24 GiB when only one client is active: the calculation uses the installed concurrency limit. Admission commits the grant in the ledger, and device allocations occur as the query executes. Source planning uses that grant to derive the chunk target.

Unless a profile sets worker_count, stream_pool_size and lane_count, each device derives these counts from A_d and S_d.

Resource	Count	Scope	Purpose
Worker threads	A_d + 1	Device	Pool workers, one kept free for lifecycle jobs.
Lifecycle job permits	A_d	Device	Context setup and query execution jobs. Release cleanup has its own A_d.
Execution job permits	A_d * S_d	Device	Lane and source-read jobs.
Stream pool	A_d * (1 + S_d)	Device	A_d root and A_d * S_d auxiliary streams, each created on first use.
Attempt lanes	A_d	Device	One index per admitted attempt (lane_count).
Auxiliary slots	S_d	Attempt	Permits behind each execution lane and source read.
Native fork streams	S_d + 1	Attempt	Streams for libcudf's fork_streams, one reserved for the root stream.
Source reads	max_source_reads_in_flight, default 2	Attempt	A value above 1 enables source-read prefetch.
Source I/O threads	64	Process	KvikIO pool shared by local and S3 reads; KVIKIO_NTHREADS overrides.

The allocation path

The grant authorizes a byte limit. Physical buffers are obtained as operations allocate. Unused grant capacity is still committed in the device ledger.

DeviceMemoryResource is the allocation interface bound to GPU work. The query domain checks allocations against its grant and obtains storage through the device's RMM memory resource. RMM is RAPIDS' memory-management library. A pool can reuse GPU allocations across operations.

The grant covers allocations in device VRAM. I/O bounce buffers and other host staging occupy system RAM, outside this GPU cap. A host buffer pinned for DMA is not charged as a device allocation.

The query domain's allocator alone enforces the query cap shared by all execution lanes. An allocation exceeding the cap returns a typed failure at its source. Chunking can reduce live intermediates. When the cap rejects an allocation while a source chunk is read, staged through the filters and projections that run on it, or partitioned for an aggregate, the chunk splits in half by row group and runs again, up to the configured retry attempts. An input, hash table or result that cannot fit can still fail. A projection of a source larger than the cap fails at the last chunk read, because the device result is complete before egress copies it to the client. Native execution does not provide general disk spilling.

Completion and resident data

An output may remain on the GPU for another native consumer. Its resident lease retains the backing buffers and completion dependencies through the last borrow. GpuDataFrame records placement and retained owners, while GpuResultHandle exposes a resident result with its completion dependencies. Derived frames track surviving columns using ColumnMemoryIdentity. A downstream projection preserves that lease while referenced columns survive, releasing transitive leases when only fresh columns remain.

Source and graph caches retain reusable data under their own device-ledger charges. Cached allocations remain accounted for within the device ledger. Output and resident facts carry scope, occurrence, schema and provenance.

Cancellation halts new task submissions while active GPU operations run to completion. ExecutionCancellation is a token with an optional deadline. The engine checks it at native boundaries (the next source chunk, task, operator or materialization step) and does not interrupt a CUDA call already running. When the caller drops the result stream, the attempt is cancelled without blocking the caller's runtime, and the terminal outcome records Dropped, Cancelled or Panicked. The chunk coordinator joins its lane jobs and synchronizes their readiness events before the lane leases return. A stream lease returns to the pool only after the cleanup thread synchronizes it. Flight connection loss takes the same path through a per-statement token.

The attempt releases its grant once its outcome is terminal and no fragment is running. The end of one fragment's stream does not release the grant. Retained outputs and pending GPU work must settle before their resources can be reused. At release, the attempt's cleanup stream is synchronized and its memory domain sampled. A failed synchronization, a persistent CUDA fault, or memory still charged to the attempt closes admission on that device. A cleanup callback dropped before it ran, or a domain sample that fails on a healthy device, leaves fault isolation unknown, and the backend then rejects new GPU work. A failed fence inside a task is recorded in the attempt report and does not change admission by itself.

Shared interop final cleanup follows the algeon-interop lease contract. cuDF records a failed final read fence or incomplete cleanup through device-health quarantine, which rejects new device work. Retained producer allocations remain charged to their memory domain for process lifetime when completion cannot be proved; cleanup does not report those bytes as reclaimed. Borrowed storage with no retained owner requires a successful fallback before its borrow ends.

Runtime observations

Device observations

Three sources describe device memory, and their scopes differ. The query domain reports its current and peak allocation, allocation count and failures for one grant. Each CUDA async pool reports reserved and used bytes with their high-water marks. rmm::available_device_memory() reports the CUDA-visible free and total bytes of the device. These sources, the device profile and the ledger supply every admission input, independent of the nvml feature.

The optional nvml feature adds GPU and memory-controller utilization, the NVML framebuffer total, reserved, used and free bytes, and the startup model, UUID and physical total. These values are diagnostics. Admission, grants and resident accounting do not read them, and CUDA totals are not copied into NVML fields. An unsampled NVML value is None. nvml_status reports FeatureDisabled when the feature was not compiled and Unavailable when initialization, device lookup or sampling failed. The CUDA sample keeps its own cuda_status. install_telemetry controls both CUDA and NVML sampling for a device.

Reports and metrics

Reports, TSV, DataFusion metrics and QueryMetrics are projections of one runtime observation snapshot, in which allocation failures retain their typed source and cleanup disposition. Metric names in NATIVE_ALGEON_DATAFUSION_METRIC_NAMES describe the metric meaning and unit, and are updated whenever the underlying definitions change.