Skip to main content

HDBSCAN

UDTF: cuml_hdbscan

Official cuML reference: Python API

cuML HDBSCAN fits density-based clusters and returns query-local labels and membership probabilities.

Quickstart​

Register the relations referenced by the parenthesized SELECT clauses below. For metadata validation, the descriptor names input_vectors.

SELECT id, cluster_id, probability
FROM cuml_hdbscan(
  input => (SELECT id, d0, d1 FROM t),
  min_cluster_size => 5
)
ORDER BY row_ordinal;

Inputs​

Each relation argument is a parenthesized SELECT subquery. Metadata validation resolves a registered table or view for the same role without scanning its rows. See ML Inputs for the ID, Float32 feature, null, finite-value, and runtime-dimension contract.

RoleRequiredValidation referenceDescription
inputyestableDense Float32 rows fitted and transformed in this statement.

Vector element types​

Element typeValid metrics
Float32Not applicable

Arguments and options​

Scalar SQL arguments​

ArgumentTypeRequiredDescription
min_samplesintegernoMinimum samples in a core-distance neighborhood.
min_cluster_sizeintegernoMinimum size of a selected cluster.
max_cluster_sizeintegernoMaximum cluster size; zero disables the bound.
cluster_selection_epsilonnumbernoNonnegative threshold for cluster selection.
alphanumbernoPositive mutual-reachability distance scale.
allow_single_clusterbooleannoAllow one cluster to cover all input rows.
cluster_selection_methodenum ("eom", "leaf")noSelect clusters by excess of mass or leaf nodes.

SQL value argument schemas​

ArgumentRequiredLiteral shapeDefaultConstraintsDescription
allow_single_clusternobooleanfalseAllow one cluster to cover all input rows.
alphanonumber1greater than 0Positive mutual-reachability distance scale.
cluster_selection_epsilonnonumber0minimum 0Nonnegative threshold for cluster selection.
cluster_selection_methodnostring"eom"one of "eom", "leaf"Select clusters by excess of mass or leaf nodes.
max_cluster_sizenointeger0minimum 0; maximum 2147483647Maximum cluster size; zero disables the bound.
min_cluster_sizenointeger5minimum 2; maximum 2147483647Minimum size of a selected cluster.
min_samplesnointeger5minimum 1; maximum 2147483647Minimum samples in a core-distance neighborhood.

Vector binding shapes​

Each relation subquery projects a non-null id and either non-null Float32 feature columns in dimension order or one non-null vector list column of non-null Float32 values. For classification, training also projects non-null Int32 label; regression requires non-null finite Float32 label. The predict relation omits it.

The list-column shape excludes other feature columns. See ML Inputs for the allowed list containers and runtime checks.

Output​

ColumnTypeNullableDescription
row_ordinalUInt64noEvaluated input row position, including when IDs repeat. Follows the input's top-level ORDER BY on output columns; unspecified without one.
idsame_as_input.idnoLogical input ID.
cluster_idInt64noSelected cluster ID; -1 indicates noise.
probabilityFloat32noMembership probability for the selected cluster.

Concrete schemas are call-specific. Run gpu_validate_call against registered relations to inspect the output schema after the actual ID types and literal options are validated.

Examples​

This example uses the EMBER2024 demo dataset.

Which missed Win32 files should an analyst group for batch review?​

A security analyst needs to organize 3,225 Win32 samples that initially escaped antivirus detection. This query projects each sample's 512 byte and byte-entropy histogram features (f7 through f518) to 16 PCA components, then fits HDBSCAN with min_cluster_size = 30 and min_samples = 10. The family labels do not enter either fit; SQL joins them after clustering to summarize the resulting groups.

The capture contains seven clusters and 508 rows assigned to noise (cluster_id = -1). HDBSCAN keeps those noise rows in the output. Cluster IDs are local to this fit. The 930-file cluster contains 736 snackarcin files, 193 opensupdater files, and one file without a family label. The 1,005-file cluster has 734 family-labeled files, 547 of them malicord. These family counts describe the families present within each fitted group. All challenge files are positive examples, so this cohort cannot measure a false-positive rate.

The SQL returns one row per cluster. example_sha256 is the minimum SHA-256 within that cluster's most common labeled family. It provides an audit link; the query does not compute a representative sample. The complete capture preserves the full SHA values.

Cluster ID (capture-local)FilesFamily-labeled filesMost common familyFiles in family
61005734malicord547
1930929snackarcin736
-1508246fragtor23
3451174fragtor28
5146123doenerium110
49157innobundle14
05959snackarcin46
2353fragtor2

Download the JSON capture, including the SQL, all eight result rows, query ID, and execution conditions.

WITH reduced AS (
  SELECT id, pc_0, pc_1, pc_2, pc_3, pc_4, pc_5, pc_6, pc_7,
         pc_8, pc_9, pc_10, pc_11, pc_12, pc_13, pc_14, pc_15
  FROM cuvs_pca(
    input => (
      SELECT v.id, v.histogram AS vector
      FROM ember_research_vectors_challenge v
      JOIN ember_metadata_challenge m ON m.id = v.id
      WHERE m.file_type = 'Win32'
      ORDER BY v.id
    ),
    n_components => 16
  )
), assignments AS (
  SELECT id, cluster_id
  FROM cuml_hdbscan(
    input => (SELECT * FROM reduced ORDER BY id),
    min_cluster_size => 30,
    min_samples => 10
  )
), family_counts AS (
  SELECT a.cluster_id, m.family, COUNT(*) AS files,
         MIN(m.sha256) AS example_sha256
  FROM assignments a
  JOIN ember_metadata_challenge m ON m.id = a.id
  GROUP BY a.cluster_id, m.family
), ranked AS (
  SELECT *,
         SUM(files) OVER (PARTITION BY cluster_id) AS cluster_files,
         SUM(CASE WHEN family <> '' THEN files ELSE 0 END)
           OVER (PARTITION BY cluster_id) AS family_labeled_files,
         ROW_NUMBER() OVER (
           PARTITION BY cluster_id
           ORDER BY CASE WHEN family = '' THEN 1 ELSE 0 END,
                    files DESC, family
         ) AS family_rank
  FROM family_counts
)
SELECT cluster_id, cluster_files, family_labeled_files,
       family AS most_common_family, files AS most_common_family_files,
       example_sha256
FROM ranked
WHERE family_rank = 1
ORDER BY cluster_files DESC, cluster_id;

The query orders the input by id, which fixes the input sequence represented by this capture. It does not guarantee identical fitted memberships or label numbers across hardware and software environments. The observed execution used a dedicated EMBER server and had partial_native disposition: PCA and HDBSCAN ran on the GPU; the statement also included CPU window processing. The saved runtime observation identifies this execution path.

Limits​

  • Validation resolves named tables or views and reads schemas only; it does not execute relation scans or GPU work.
  • Execution relation arguments require parenthesized subqueries; dry-run validation accepts registered named relations only.
  • Only Float32 features are supported; L2SqrtExpanded distance and brute-force kNN are fixed.
  • Empty input returns an empty result. Other inputs require min_samples < rows, min_cluster_size <= rows, and max_cluster_size <= rows; row-dependent conditions are checked at execution.
  • Cluster ID -1 denotes noise. The fit and output are query-local.

To dry-run validate relation metadata, column types, and options without execution, see gpu_validate_call.

On this page