transforms

transforms#

galactic.transforms.ai_column(self, new_column, prompt, depends_on=list[str], normalize=['strip'], system_prompt=None)[source]#

Use OpenAI’s API to generate a new column based on an existing column.

Parameters:

  • new_column (str) – Name of the new column.

  • prompt (str) – Prompt template (Jinja2) to use for the API request.

  • depends_on (list[str], optional) – List of fields needed to fill in the template, defaults to an empty list.

  • normalize (list[str], optional) – List of string methods to apply for normalization, defaults to [“strip”, “lower”]

  • system_prompt (str, optional) – System instructions (optional, same for every request), defaults to None

Returns:

Modified GalacticDataset instance.

Return type:

GalacticDataset

Raises:

ValueError – If a specified field is not found in the dataset or if the new_column already exists in the dataset.

dataset = GalacticDataset(…)
dataset.ai_column(new_column='ai_column', prompt='{{column1}} is related to {{column2}}', depends_on=['column1', 'column2'])
galactic.transforms.trim_whitespace(self, fields, inplace=True)[source]#

Trim Unicode-defined whitespace at the beginning and end of specified fields.

Parameters:

  • fields (Sequence[str]) – List of fields to trim.

  • inplace (bool, optional) – Whether to perform the operation in-place, defaults to True

Returns:

Modified GalacticDataset instance.

Return type:

GalacticDataset

dataset = GalacticDataset(…)
dataset.trim_whitespace(fields=['column1', 'column2'])
galactic.transforms.unicode_normalize(self, fields, form='NFC', inplace=True)[source]#

Apply Unicode normalization to specified fields. This is useful as a pre-processing step for ML training.

Parameters:

  • fields (Sequence[str]) – List of fields to normalize.

  • form (str, optional) – Unicode normalization form, defaults to “NFC”

  • inplace (bool, optional) – Whether to perform the operation in-place, defaults to True

Returns:

Modified GalacticDataset instance if inplace is True, else a new instance.

Return type:

GalacticDataset

dataset = GalacticDataset(…)
dataset.unicode_normalize(fields=['column1', 'column2'], form='NFC')
Contents