clustering

galactic.clustering.ai_label_clusters(self, new_column, context_fields, cluster_field='__cluster', embedding_field='__embedding', n_examples=10, selection='random', prompt=None)

Labels the clusters using AI based on the given fields and prompt template.

# Example usage:
ds.ai_label_clusters(fields=['field1', 'field2'], selection='nearest')

Parameters:

• fields – List of fields used to identify clusters.

• new_column (str) – Optional. Name of the new column where the cluster labels will be stored. Defaults to ‘__cluster_label’.

• n_examples (int) – Optional. Number of examples to consider for labeling. Defaults to 10.

• selection (Literal['random', 'nearest']) – Optional. Strategy to select examples for labeling. Can be ‘random’ or ‘nearest’. Defaults to ‘random’.

• embedding_field (str) – Optional. Name of the embedding field to be used. Defaults to ‘__embedding’.

• prompt (Optional[str]) – Optional. Jinja2 template string to be used as the prompt for labeling.

• context_fields (list[str]) –

• cluster_field (str) –

Returns:

The modified object with labeled clusters.

galactic.clustering.cluster(self, n_clusters=None, method='kmeans', embedding_field='__embedding', cluster_field='__cluster', overwrite=False, **kwargs)

Cluster the dataset using the specified method.

# Cluster your dataset into 10 clusters using minibatch k-means
ds.cluster(n_clusters=10)

Parameters:

• n_clusters (Optional[int]) – Required. The number of clusters to form.

• method (Literal['kmeans', 'minibatch_kmeans', 'hdbscan']) – Optional. The clustering method to use. Default = kmeans.

• embedding_field (str) – Optional. Specify a name for the field to use for clustering. Default = __embedding.

• kwargs – Optional. Additional keyword arguments to pass to the clustering algorithm.

• cluster_field (str) –

• overwrite (bool) –

galactic.clustering.get_cluster_info(self, n_neighbors=3, cluster_field='__cluster', embedding_field='__embedding', context_fields=[], truncate_fields=250, verbose=True)

Retrieves information regarding clusters and their nearest neighbors.

# Example usage:
ds.get_cluster_info(n_neighbors=5, field='title')

Parameters:

• n_neighbors (int) – Number of nearest neighbors to be retrieved for each cluster. Defaults to 3.

• field – Optional specific field to be printed from the neighbors.

• cluster_field (str) –

• embedding_field (str) –

• context_fields (list[str]) –

• truncate_fields (int) –

• verbose (bool) –

Raises:

ValueError – If cluster() or get_embeddings() methods are not called prior to calling this method.

galactic.clustering.get_duplicates(cluster, cluster_center, threshold, emb_matrix=None, similarities=None, embedding_field='__embedding', dedup_strategy='random')

Retrieves duplicates within a specified cluster based on a threshold.

# Example usage:
duplicates = get_duplicates(cluster=ds, threshold=0.8, strategy='nearest')

Parameters:

• cluster (Dataset) – The dataset cluster to be analyzed for duplicates.

• threshold (float) – Similarity threshold to consider two items as duplicates.

• strategy – Strategy to select duplicates, can be ‘random’, ‘nearest’, or ‘furthest’. Defaults to ‘random’.

• cluster_center (numpy.ndarray) –

• emb_matrix (numpy.ndarray | None) –

• similarities (numpy.ndarray | None) –

• embedding_field (str) –

• dedup_strategy (Literal['random', 'nearest', 'furthest']) –

Raises:

ValueError – If an unknown strategy is provided.

Returns:

List of identified duplicate items within the cluster.

galactic.clustering.recompute_cluster_centers(self, cluster_field='__cluster', embedding_field='__embedding', method='centroid')

Re-compute centers and store them in self.cluster_centers. There are two main reasons to do this: 1. You’ve deduplicated or sparsified clusters, so their centers have moved, or 2. You’ve re-loaded the data from file, so the centers are no longer in memory.

Parameters:

• cluster_field (str) –

• embedding_field (str) –

• method (Literal['centroid', 'medoid']) –

galactic.clustering.remove_cluster(self, cluster_field, cluster_id)

Remove a cluster from the dataset.

Parameters:

• cluster_field (str) –

• cluster_id (int) –

galactic.clustering.remove_prototypes(self, target_retention=0.75, cluster_field='__cluster', embedding_field='__embedding', inplace=True)

Apply the ‘SSL prototypes’ strategy from D4 (https://arxiv.org/pdf/2308.12284.pdf), which removes the most prototypical examples from each cluster (i.e. those closest to the center) until the target retention rate is reached.

Parameters:

• target_retention (float) –

• cluster_field (str) –

• embedding_field (str) –

• inplace (bool) –

galactic.clustering.semdedup(self, target_retention=0.8, threshold=None, embedding_field='__embedding', cluster_field='__cluster', num_tuning_clusters=3, dedup_strategy='random', inplace=True)

Remove semantic near-duplicates from the dataset.

Parameters:

• target_retention (float | None) –

• threshold (float | None) –

• embedding_field (str) –

• cluster_field (str) –

• num_tuning_clusters (int) –

• dedup_strategy (Literal['random', 'nearest', 'furthest']) –

galactic.clustering.tune_threshold(cluster, cluster_center, target_retention, tol=0.01, max_iter=30, embedding_field='__embedding', dedup_strategy='random')

Tunes the threshold for identifying duplicates within a cluster to achieve a target retention rate.

# Example usage:
threshold = tune_threshold(cluster=ds, target_retention=0.9)

Parameters:

• cluster (Dataset) – The dataset cluster to be analyzed.

• target_retention (float) – Target retention rate to achieve.

• tol (float) – Tolerance for the difference between achieved and target retention rate. Defaults to 0.01.

• max_iter (int) – Maximum number of iterations for tuning. Defaults to 30.

• cluster_center (numpy.ndarray) –

• embedding_field (str) –

• dedup_strategy (Literal['random', 'nearest', 'furthest']) –

Returns:

The tuned threshold value.