Skip to main content

HDBSCAN

UDTF: cuml_hdbscan

Official cuML reference: Python API

cuML HDBSCAN fits density-based clusters and returns query-local labels and membership probabilities.

Quickstart​

Register the relations referenced by the parenthesized SELECT clauses below. For metadata validation, the descriptor names input_vectors.

SELECT id, cluster_id, probability
FROM cuml_hdbscan(
input => (SELECT id, d0, d1 FROM t),
min_cluster_size => 5
)
ORDER BY row_ordinal;

Inputs​

Each relation argument is a parenthesized SELECT subquery. Metadata validation resolves a registered table or view for the same role without scanning its rows. See ML Inputs for the ID, feature element, Float64 narrowing, null, finite-value, and runtime-dimension contract.

RoleRequiredValidation referenceDescription
inputyestableDense feature rows evaluated by the call.

Vector element types​

Feature values must be Float32 or Float64.

Arguments and options​

Scalar SQL arguments​

ArgumentTypeRequiredDefaultConstraintsDescription
allow_single_clusterbooleannofalseAllow one cluster to cover all input rows.
alphanumberno1greater than 0Positive mutual-reachability distance scale.
cluster_selection_epsilonnumberno0minimum 0Nonnegative threshold for cluster selection.
cluster_selection_methodstringno"eom"one of "eom", "leaf"Select clusters by excess of mass or leaf nodes.
max_cluster_sizeintegerno0minimum 0; maximum 2147483647Maximum cluster size; zero disables the bound.
min_cluster_sizeintegerno5minimum 2; maximum 2147483647Minimum size of a selected cluster.
min_samplesintegerno5minimum 1; maximum 2147483647Minimum samples in a core-distance neighborhood.

Vector binding shapes​

Each relation subquery projects an id and either feature columns in dimension order or one list column, with the element types listed under Vector element types. Nullable field types are accepted. A null ID, feature value, list row, or list element fails the query.

The list-column shape excludes other feature columns. See ML Inputs for the allowed list containers and runtime checks.

Output​

ColumnTypeNullableDescription
row_ordinalUInt64noEvaluated input row position, including when IDs repeat. Follows the input's top-level ORDER BY on output columns; unspecified without one.
idsame_as_input.idnoLogical input ID.
cluster_idInt64noSelected cluster ID; -1 indicates noise.
probabilityFloat32noMembership probability for the selected cluster.

The output schema depends on the call. Run gpu_validate_call against registered relations to inspect the schema that the validated ID types and literal options produce.

Examples​

This example uses the EMBER2024 demo dataset.

Which missed Win32 files should an analyst group for batch review?​

The 3,225 Win32 samples in this cohort initially escaped antivirus detection. This query projects each sample's 512 byte and byte-entropy histogram features (f7 through f518) to 16 PCA components, then fits HDBSCAN with min_cluster_size = 30 and min_samples = 10. The family labels do not enter either fit; SQL joins them after clustering to summarize the resulting groups.

The capture contains seven clusters and 508 rows assigned to noise (cluster_id = -1). HDBSCAN keeps those noise rows in the output. Cluster IDs are local to this fit. The 930-file cluster contains 736 snackarcin files, 193 opensupdater files, and one file without a family label. All challenge files are positive examples, so this cohort cannot measure a false-positive rate.

Cluster ID (capture-local)FilesFamily-labeled filesMost common familyFiles in family
61005734malicord547
1930929snackarcin736
-1508246fragtor23
3451174fragtor28
5146123doenerium110
49157innobundle14
05959snackarcin46
2353fragtor2
WITH reduced AS (
SELECT id, pc_0, pc_1, pc_2, pc_3, pc_4, pc_5, pc_6, pc_7,
pc_8, pc_9, pc_10, pc_11, pc_12, pc_13, pc_14, pc_15
FROM cuvs_pca(
input => (
SELECT v.id, v.histogram AS vector
FROM ember_research_vectors_challenge v
JOIN ember_metadata_challenge m ON m.id = v.id
WHERE m.file_type = 'Win32'
ORDER BY v.id
),
n_components => 16
)
), assignments AS (
SELECT id, cluster_id
FROM cuml_hdbscan(
input => (SELECT * FROM reduced ORDER BY id),
min_cluster_size => 30,
min_samples => 10
)
), family_counts AS (
SELECT a.cluster_id, m.family, COUNT(*) AS files,
MIN(m.sha256) AS example_sha256
FROM assignments a
JOIN ember_metadata_challenge m ON m.id = a.id
GROUP BY a.cluster_id, m.family
), ranked AS (
SELECT *,
SUM(files) OVER (PARTITION BY cluster_id) AS cluster_files,
SUM(CASE WHEN family <> '' THEN files ELSE 0 END)
OVER (PARTITION BY cluster_id) AS family_labeled_files,
ROW_NUMBER() OVER (
PARTITION BY cluster_id
ORDER BY CASE WHEN family = '' THEN 1 ELSE 0 END,
files DESC, family
) AS family_rank
FROM family_counts
)
SELECT cluster_id, cluster_files, family_labeled_files,
family AS most_common_family, files AS most_common_family_files,
example_sha256
FROM ranked
WHERE family_rank = 1
ORDER BY cluster_files DESC, cluster_id;

The query orders the input by id, which fixes the input sequence represented by this capture. It does not guarantee identical fitted memberships or label numbers across hardware and software environments.

The saved capture retains the SQL, all eight result rows, query ID and execution conditions. The observed execution used a dedicated EMBER server and had partial_native disposition: PCA and HDBSCAN ran on the GPU. The plan also included CPU window processing.

Limits​

  • L2SqrtExpanded distance and brute-force kNN are fixed.
  • Float64 features, as wide columns or list elements, are narrowed to Float32 on the device before training or inference. Each value rounds to the nearest Float32, which keeps about seven significant digits; a value beyond the Float32 range becomes infinite and fails the query.
  • Empty input returns an empty result. Other inputs require min_samples < rows, min_cluster_size <= rows, and max_cluster_size <= rows; row-dependent conditions are checked at execution.
  • Cluster ID -1 denotes noise. The fit and output are query-local.