taggers

taggers#

galactic.taggers.ai_tagger(self, field, tags, prompt=None, backend='openai', allow_not_sure=False)[source]#

Use OpenAI’s API or a HF zero-shot classifier to generate a new column based on an existing column. :param name: Name of the new column. :type name: str :type field: str :param field: Name of the field to use for classification. Can be None if backend is ‘embeddings’. :type field: str :param classes: Classes to classify into. Can be just a list of labels, or a dict of label: description. :type classes: List[str] or dict[str, str] :param If provided: :param descriptions will be used for a) prompting API model if backend = ‘openai’: :param b) embedding if backend = “embeddings”: :param : :param c) zero-shot classification if backend = “huggingface”.: :type backend: :param backend: Which backend to use. Currently supports ‘embeddings’, ‘openai’ and ‘huggingface’. :type backend: str

Parameters:

  • field (str) –

  • tags (list[str] | dict[str, str]) –

  • prompt (str | None) –

  • allow_not_sure (bool) –

galactic.taggers.calc_perplexity(self, field, model='kenlm', language='en', dataset='wikipedia')[source]#

Calculate the perplexity-per-byte of the specified field.

Note

You must install KenLM to calculate perplexity using the KenLM model from kpu/kenlm. To install, run pip install https://github.com/kpu/kenlm/archive/master.zip.

# Replace "field_name1"
ds.calc_perplexity(field="field_name1")
Parameters:

  • field (str) –

  • model (str) –

  • language (str | None) –

  • dataset (str | None) –

galactic.taggers.count_tokens(self, fields, tokenizer=None)[source]#

Count the number of tokens in the specified fields.

# Replace "field_name1" and "field_name2" with your fields
ds.count_tokens(fields=["field_name1, field_name2"])
Parameters:

  • fields (List[str]) – List of fields to count tokens in.

  • tokenizer (Callable) – Tokenizer function to use. Defaults to None, which uses bytes.

galactic.taggers.detect_language(self, field, method='fasttext')[source]#

Detect the language of the specified field.

Parameters:

  • field (str) –

  • method (Literal['fasttext', 'langdetect']) –

galactic.taggers.detect_pii(self, fields)[source]#

Detect personally identifiable information in the specified fields. :type fields: Sequence[str] :param fields: List of fields to detect PII in. :type fields: List[str] :param Currently only supports “email”: :param “phone”: :param and “credential”.:

Parameters:

fields (Sequence[str]) –

galactic.taggers.tag_string(self, fields, values, tag)[source]#

Tag data containing a particular string or list of strings in the specified field.

Parameters:

  • fields (Sequence[str]) –

  • values (Sequence[str]) –

  • tag (str) –

Contents