loaders#
- galactic.loaders.from_csv(cls, path)[source]#
Loads a dataset from a CSV file.
# Example usage: ds = GalacticDataset.from_csv('path_to_file.csv')
- Parameters:
path (str) – The path to the CSV file.
- Returns:
An instance of the class populated with the dataset from the CSV file.
- Return type:
cls
- galactic.loaders.from_hugging_face(cls, path, split, config_name=None, **kwargs)[source]#
Loads a dataset from the Hugging Face hub.
# Example usage: ds = GalacticDataset.from_hugging_face('dataset_path', 'split', 'config_name')
- Parameters:
path (str) – The path to the dataset on the Hugging Face hub.
split (str) – The specific split of the dataset to load.
config_name (Optional[str]) – The name of the configuration to load, if applicable. Defaults to None.
kwargs – Additional keyword arguments passed to datasets.load_dataset().
- Returns:
An instance of the class populated with the dataset from the Hugging Face hub.
- Return type:
cls
- galactic.loaders.from_hugging_face_stream(cls, path, split, config_name=None, filters=None, dedup_fields=None, max_samples=200000, **kwargs)[source]#
Loads a dataset from the Hugging Face hub, streaming the data from disk.
# Example usage: ds = GalacticDataset.from_hugging_face_stream('dataset_path', 'split', filters=[filter_func], dedup_fields=['field1', 'field2'])
- Parameters:
path (str) – The path to the dataset on the Hugging Face hub.
split (str) – The specific split of the dataset to load.
config_name (Optional[str]) – The name of the configuration to load, if applicable. Defaults to None.
filters (Optional[list[Callable[[dict], bool]]]) – A list of filter functions to apply to the stream. Each filter function should take a dictionary and return a boolean. Defaults to None.
dedup_fields (Optional[list[str]]) – A list of fields to use for deduplication via a Bloom filter. Defaults to None.
max_samples (Optional[int]) – The maximum number of samples to load. Defaults to 200000.
kwargs – Additional keyword arguments passed to datasets.load_dataset().
- Returns:
An instance of the class populated with the streamed dataset from the Hugging Face hub.
- Return type:
cls
- galactic.loaders.from_jsonl(cls, path)[source]#
Loads a dataset from a JSONL file.
# Example usage: ds = GalacticDataset.from_jsonl('path_to_file.jsonl')
- Parameters:
path (str) – The path to the JSONL file.
- Returns:
An instance of the class populated with the dataset from the JSONL file.
- Return type:
cls
- galactic.loaders.from_pandas(cls, df)[source]#
Loads a dataset from a Pandas DataFrame.
# Example usage: ds = GalacticDataset.from_pandas(dataframe)
- Parameters:
df (pd.DataFrame) – The Pandas DataFrame containing the dataset.
- Returns:
An instance of the class populated with the dataset from the DataFrame.
- Return type:
cls
- galactic.loaders.from_parquet(cls, path)[source]#
Loads a dataset from a Parquet file.
# Example usage: ds = GalacticDataset.from_parquet('path_to_file.parquet')
- Parameters:
path (str) – The path to the Parquet file.
- Returns:
An instance of the class populated with the dataset from the Parquet file.
- Return type:
cls
- galactic.loaders.save(self, path, overwrite=False)[source]#
Saves the dataset as a CSV or JSONL file.
# Example usage: ds.save('path_to_file.csv', overwrite=True)
- Parameters:
path (str) – The path where the dataset will be saved. The format is inferred from the file extension (either .csv or .jsonl).
overwrite (bool) – Whether to overwrite the file if it already exists. Defaults to False.
- Raises:
ValueError – If the path already exists and overwrite is False, or if the file format is unsupported.
- Return type:
None