Skip to main content

​
BaseGenerator

class BaseGenerator(ABC, IterableDataset):
Abstract base class for data generators. This class defines the interface for generating synthetic datapoints. Concrete implementations should provide specific generation strategies.

​
init

def __init__(
    self,
    seed: int = 42,
    buffer: int = 20,
    cache: Union[str, Path, None] = None,
    data_path: Union[str, Path, None] = None,
    **kwargs
):
Initialize the base generator. Parameters:
  • seed (int): Random seed for reproducibility. (default: :obj:42) (default: 42)
  • buffer (int): Amount of DataPoints to be generated when the iterator runs out of DataPoints in data. (default: :obj:20)
  • cache (Union[str, Path, None]): Optional path to save generated datapoints during iteration. If None is provided, datapoints will be discarded every 100 generations.
  • data_path (Union[str, Path, None]): Optional path to a JSONL file to initialize the dataset from. **kwargs: Additional generator parameters.

​
aiter

def __aiter__(self):
Async iterator that yields datapoints dynamically. If a data_path was provided during initialization, those datapoints are yielded first. When self._data is empty, 20 new datapoints are generated. Every 100 yields, the batch is appended to the JSONL file or discarded if cache is None. Yields: DataPoint: A single datapoint.

​
iter

def __iter__(self):
Synchronous iterator for PyTorch IterableDataset compatibility. If a data_path was provided during initialization, those datapoints are yielded first. When self._data is empty, 20 new datapoints are generated. Every 100 yields, the batch is appended to the JSONL file or discarded if cache is None. Yields: DataPoint: A single datapoint.

​
sample

def sample(self):
Returns: DataPoint: The next DataPoint. Note: This method is intended for synchronous contexts. Use ‘async_sample’ in asynchronous contexts to avoid blocking or runtime errors.

​
save_to_jsonl

def save_to_jsonl(self, file_path: Union[str, Path]):
Saves the generated datapoints to a JSONL (JSON Lines) file. Each datapoint is stored as a separate JSON object on a new line. Parameters:
  • file_path (Union[str, Path]): Path to save the JSONL file.
Note:
  • Uses self._data, which contains the generated datapoints.
  • Appends to the file if it already exists.
  • Ensures compatibility with large datasets by using JSONL format.

​
flush

def flush(self, file_path: Union[str, Path]):
Flush the current data to a JSONL file and clear the data. Parameters:
  • file_path (Union[str, Path]): Path to save the JSONL file.
Note:
  • Uses save_to_jsonl to save self._data.

​
_init_from_jsonl

def _init_from_jsonl(self, file_path: Path):
Load and parse a dataset from a JSONL file. Parameters:
  • file_path (Path): Path to the JSONL file.
Returns: List[Dict[str, Any]]: A list of datapoint dictionaries.