camel.storages.vectordb_storages package#
Submodules#
camel.storages.vectordb_storages.base module#
- class camel.storages.vectordb_storages.base.BaseVectorStorage[source]#
Bases:
ABC
An abstract base class for vector storage systems.
- abstract add(records: List[VectorRecord], **kwargs: Any) None [source]#
Saves a list of vector records to the storage.
- Parameters:
records (List[VectorRecord]) – List of vector records to be saved.
**kwargs (Any) – Additional keyword arguments.
- Raises:
RuntimeError – If there is an error during the saving process.
- abstract property client: Any#
Provides access to the underlying vector database client.
- abstract delete(ids: List[str], **kwargs: Any) None [source]#
Deletes a list of vectors identified by their IDs from the storage.
- Parameters:
ids (List[str]) – List of unique identifiers for the vectors to be deleted.
**kwargs (Any) – Additional keyword arguments.
- Raises:
RuntimeError – If there is an error during the deletion process.
- get_payloads_by_vector(vector: List[float], top_k: int) List[Dict[str, Any]] [source]#
Returns payloads of top k vector records that closest to the given vector.
This function is a wrapper of BaseVectorStorage.query.
- Parameters:
vector (List[float]) – The search vector.
top_k (int) – The number of top similer vectors.
- Returns:
- A list of vector payloads retrieved
from the storage based on similarity to the query vector.
- Return type:
List[List[Dict[str, Any]]]
- abstract query(query: VectorDBQuery, **kwargs: Any) List[VectorDBQueryResult] [source]#
Searches for similar vectors in the storage based on the provided query.
- Parameters:
query (VectorDBQuery) – The query object containing the search vector and the number of top similar vectors to retrieve.
**kwargs (Any) – Additional keyword arguments.
- Returns:
- A list of vectors retrieved from the
storage based on similarity to the query vector.
- Return type:
List[VectorDBQueryResult]
- abstract status() VectorDBStatus [source]#
Returns status of the vector database.
- Returns:
The vector database status.
- Return type:
- class camel.storages.vectordb_storages.base.VectorDBQuery(query_vector: List[float], top_k: int)[source]#
Bases:
BaseModel
Represents a query to a vector database.
- query_vector#
The numerical representation of the query vector.
- Type:
List[float]
- top_k#
The number of top similar vectors to retrieve from the database. (default:
1
)- Type:
int, optional
- model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#
A dictionary of computed field names and their corresponding ComputedFieldInfo objects.
- model_config: ClassVar[ConfigDict] = {}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_fields: ClassVar[Dict[str, FieldInfo]] = {'query_vector': FieldInfo(annotation=List[float], required=True), 'top_k': FieldInfo(annotation=int, required=False, default=1)}#
Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.
This replaces Model.__fields__ from Pydantic V1.
- query_vector: List[float]#
The numerical representation of the query vector.
- top_k: int#
The number of top similar vectors to retrieve from the database.
- class camel.storages.vectordb_storages.base.VectorDBQueryResult(*, record: VectorRecord, similarity: float)[source]#
Bases:
BaseModel
Encapsulates the result of a query against a vector database.
- record#
The target vector record.
- Type:
- similarity#
The similarity score between the query vector and the record.
- Type:
float
- classmethod create(similarity: float, vector: List[float], id: str, payload: Dict[str, Any] | None = None) VectorDBQueryResult [source]#
A class method to construct a VectorDBQueryResult instance.
- model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#
A dictionary of computed field names and their corresponding ComputedFieldInfo objects.
- model_config: ClassVar[ConfigDict] = {}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_fields: ClassVar[Dict[str, FieldInfo]] = {'record': FieldInfo(annotation=VectorRecord, required=True), 'similarity': FieldInfo(annotation=float, required=True)}#
Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.
This replaces Model.__fields__ from Pydantic V1.
- record: VectorRecord#
- similarity: float#
- class camel.storages.vectordb_storages.base.VectorDBStatus(*, vector_dim: int, vector_count: int)[source]#
Bases:
BaseModel
Vector database status.
- vector_dim#
The dimention of stored vectors.
- Type:
int
- vector_count#
The number of stored vectors.
- Type:
int
- model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#
A dictionary of computed field names and their corresponding ComputedFieldInfo objects.
- model_config: ClassVar[ConfigDict] = {}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_fields: ClassVar[Dict[str, FieldInfo]] = {'vector_count': FieldInfo(annotation=int, required=True), 'vector_dim': FieldInfo(annotation=int, required=True)}#
Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.
This replaces Model.__fields__ from Pydantic V1.
- vector_count: int#
- vector_dim: int#
- class camel.storages.vectordb_storages.base.VectorRecord(*, vector: List[float], id: str = None, payload: Dict[str, Any] | None = None)[source]#
Bases:
BaseModel
Encapsulates information about a vector’s unique identifier and its payload, which is primarily used as a data transfer object when saving to vector storage.
- vector#
The numerical representation of the vector.
- Type:
List[float]
- id#
A unique identifier for the vector. If not provided, an random uuid will be assigned.
- Type:
str, optional
- payload#
Any additional metadata or information related to the vector. (default:
None
)- Type:
Optional[Dict[str, Any]], optional
- id: str#
- model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#
A dictionary of computed field names and their corresponding ComputedFieldInfo objects.
- model_config: ClassVar[ConfigDict] = {}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_fields: ClassVar[Dict[str, FieldInfo]] = {'id': FieldInfo(annotation=str, required=False, default_factory=<lambda>), 'payload': FieldInfo(annotation=Union[Dict[str, Any], NoneType], required=False, default=None), 'vector': FieldInfo(annotation=List[float], required=True)}#
Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.
This replaces Model.__fields__ from Pydantic V1.
- payload: Dict[str, Any] | None#
- vector: List[float]#
camel.storages.vectordb_storages.milvus module#
- class camel.storages.vectordb_storages.milvus.MilvusStorage(vector_dim: int, url_and_api_key: Tuple[str, str], collection_name: str | None = None, **kwargs: Any)[source]#
Bases:
BaseVectorStorage
An implementation of the BaseVectorStorage for interacting with Milvus, a cloud-native vector search engine.
The detailed information about Milvus is available at: Milvus
- Parameters:
vector_dim (int) – The dimenstion of storing vectors.
url_and_api_key (Tuple[str, str]) – Tuple containing the URL and API key for connecting to a remote Milvus instance. URL maps to Milvus uri concept, typically “endpoint:port”. API key maps to Milvus token concept, for self-hosted it’s “username:pwd”, for Zilliz Cloud (fully-managed Milvus) it’s API Key.
collection_name (Optional[str], optional) – Name for the collection in the Milvus. If not provided, set it to the current time with iso format. (default:
None
)**kwargs (Any) – Additional keyword arguments for initializing MilvusClient.
- Raises:
ImportError – If pymilvus package is not installed.
- add(records: List[VectorRecord], **kwargs) None [source]#
Adds a list of vectors to the specified collection.
- Parameters:
records (List[VectorRecord]) – List of vectors to be added.
**kwargs (Any) – Additional keyword arguments pass to insert.
- Raises:
RuntimeError – If there was an error in the addition process.
- clear() None [source]#
Removes all vectors from the Milvus collection. This method deletes the existing collection and then recreates it with the same schema to effectively remove all stored vectors.
- property client: Any#
Provides direct access to the Milvus client. This property allows for direct interactions with the Milvus client for operations that are not covered by the MilvusStorage class.
- Returns:
The Milvus client instance.
- Return type:
Any
- delete(ids: List[str], **kwargs: Any) None [source]#
Deletes a list of vectors identified by their IDs from the storage. If unsure of ids you can first query the collection to grab the corresponding data.
- Parameters:
ids (List[str]) – List of unique identifiers for the vectors to be deleted.
**kwargs (Any) – Additional keyword arguments passed to delete.
- Raises:
RuntimeError – If there is an error during the deletion process.
- query(query: VectorDBQuery, **kwargs: Any) List[VectorDBQueryResult] [source]#
Searches for similar vectors in the storage based on the provided query.
- Parameters:
query (VectorDBQuery) – The query object containing the search vector and the number of top similar vectors to retrieve.
**kwargs (Any) – Additional keyword arguments passed to search.
- Returns:
- A list of vectors retrieved from the
storage based on similarity to the query vector.
- Return type:
List[VectorDBQueryResult]
- status() VectorDBStatus [source]#
Retrieves the current status of the Milvus collection. This method provides information about the collection, including its vector dimensionality and the total number of vectors stored.
- Returns:
- An object containing information about the
collection’s status.
- Return type:
camel.storages.vectordb_storages.qdrant module#
- class camel.storages.vectordb_storages.qdrant.QdrantStorage(vector_dim: int, collection_name: str | None = None, url_and_api_key: Tuple[str, str] | None = None, path: str | None = None, distance: VectorDistance = VectorDistance.COSINE, delete_collection_on_del: bool = False, **kwargs: Any)[source]#
Bases:
BaseVectorStorage
An implementation of the BaseVectorStorage for interacting with Qdrant, a vector search engine.
The detailed information about Qdrant is available at: Qdrant
- Parameters:
vector_dim (int) – The dimenstion of storing vectors.
collection_name (Optional[str], optional) – Name for the collection in the Qdrant. If not provided, set it to the current time with iso format. (default:
None
)url_and_api_key (Optional[Tuple[str, str]], optional) – Tuple containing the URL and API key for connecting to a remote Qdrant instance. (default:
None
)path (Optional[str], optional) – Path to a directory for initializing a local Qdrant client. (default:
None
)distance (VectorDistance, optional) – The distance metric for vector comparison (default:
VectorDistance.COSINE
)delete_collection_on_del (bool, optional) – Flag to determine if the collection should be deleted upon object destruction. (default:
False
)**kwargs (Any) – Additional keyword arguments for initializing QdrantClient.
Notes
If url_and_api_key is provided, it takes priority and the client will attempt to connect to the remote Qdrant instance using the URL endpoint.
If url_and_api_key is not provided and path is given, the client will use the local path to initialize Qdrant.
If neither url_and_api_key nor path is provided, the client will be initialized with an in-memory storage (“:memory:”).
- add(records: List[VectorRecord], **kwargs) None [source]#
Adds a list of vectors to the specified collection.
- Parameters:
vectors (List[VectorRecord]) – List of vectors to be added.
**kwargs (Any) – Additional keyword arguments.
- Raises:
RuntimeError – If there was an error in the addition process.
- property client: Any#
Provides access to the underlying vector database client.
- delete(ids: List[str], **kwargs: Any) None [source]#
Deletes a list of vectors identified by their IDs from the storage.
- Parameters:
ids (List[str]) – List of unique identifiers for the vectors to be deleted.
**kwargs (Any) – Additional keyword arguments.
- Raises:
RuntimeError – If there is an error during the deletion process.
- query(query: VectorDBQuery, **kwargs: Any) List[VectorDBQueryResult] [source]#
Searches for similar vectors in the storage based on the provided query.
- Parameters:
query (VectorDBQuery) – The query object containing the search vector and the number of top similar vectors to retrieve.
**kwargs (Any) – Additional keyword arguments.
- Returns:
- A list of vectors retrieved from the
storage based on similarity to the query vector.
- Return type:
List[VectorDBQueryResult]
- status() VectorDBStatus [source]#
Returns status of the vector database.
- Returns:
The vector database status.
- Return type:
Module contents#
- class camel.storages.vectordb_storages.BaseVectorStorage[source]#
Bases:
ABC
An abstract base class for vector storage systems.
- abstract add(records: List[VectorRecord], **kwargs: Any) None [source]#
Saves a list of vector records to the storage.
- Parameters:
records (List[VectorRecord]) – List of vector records to be saved.
**kwargs (Any) – Additional keyword arguments.
- Raises:
RuntimeError – If there is an error during the saving process.
- abstract property client: Any#
Provides access to the underlying vector database client.
- abstract delete(ids: List[str], **kwargs: Any) None [source]#
Deletes a list of vectors identified by their IDs from the storage.
- Parameters:
ids (List[str]) – List of unique identifiers for the vectors to be deleted.
**kwargs (Any) – Additional keyword arguments.
- Raises:
RuntimeError – If there is an error during the deletion process.
- get_payloads_by_vector(vector: List[float], top_k: int) List[Dict[str, Any]] [source]#
Returns payloads of top k vector records that closest to the given vector.
This function is a wrapper of BaseVectorStorage.query.
- Parameters:
vector (List[float]) – The search vector.
top_k (int) – The number of top similer vectors.
- Returns:
- A list of vector payloads retrieved
from the storage based on similarity to the query vector.
- Return type:
List[List[Dict[str, Any]]]
- abstract query(query: VectorDBQuery, **kwargs: Any) List[VectorDBQueryResult] [source]#
Searches for similar vectors in the storage based on the provided query.
- Parameters:
query (VectorDBQuery) – The query object containing the search vector and the number of top similar vectors to retrieve.
**kwargs (Any) – Additional keyword arguments.
- Returns:
- A list of vectors retrieved from the
storage based on similarity to the query vector.
- Return type:
List[VectorDBQueryResult]
- abstract status() VectorDBStatus [source]#
Returns status of the vector database.
- Returns:
The vector database status.
- Return type:
- class camel.storages.vectordb_storages.MilvusStorage(vector_dim: int, url_and_api_key: Tuple[str, str], collection_name: str | None = None, **kwargs: Any)[source]#
Bases:
BaseVectorStorage
An implementation of the BaseVectorStorage for interacting with Milvus, a cloud-native vector search engine.
The detailed information about Milvus is available at: Milvus
- Parameters:
vector_dim (int) – The dimenstion of storing vectors.
url_and_api_key (Tuple[str, str]) – Tuple containing the URL and API key for connecting to a remote Milvus instance. URL maps to Milvus uri concept, typically “endpoint:port”. API key maps to Milvus token concept, for self-hosted it’s “username:pwd”, for Zilliz Cloud (fully-managed Milvus) it’s API Key.
collection_name (Optional[str], optional) – Name for the collection in the Milvus. If not provided, set it to the current time with iso format. (default:
None
)**kwargs (Any) – Additional keyword arguments for initializing MilvusClient.
- Raises:
ImportError – If pymilvus package is not installed.
- add(records: List[VectorRecord], **kwargs) None [source]#
Adds a list of vectors to the specified collection.
- Parameters:
records (List[VectorRecord]) – List of vectors to be added.
**kwargs (Any) – Additional keyword arguments pass to insert.
- Raises:
RuntimeError – If there was an error in the addition process.
- clear() None [source]#
Removes all vectors from the Milvus collection. This method deletes the existing collection and then recreates it with the same schema to effectively remove all stored vectors.
- property client: Any#
Provides direct access to the Milvus client. This property allows for direct interactions with the Milvus client for operations that are not covered by the MilvusStorage class.
- Returns:
The Milvus client instance.
- Return type:
Any
- delete(ids: List[str], **kwargs: Any) None [source]#
Deletes a list of vectors identified by their IDs from the storage. If unsure of ids you can first query the collection to grab the corresponding data.
- Parameters:
ids (List[str]) – List of unique identifiers for the vectors to be deleted.
**kwargs (Any) – Additional keyword arguments passed to delete.
- Raises:
RuntimeError – If there is an error during the deletion process.
- query(query: VectorDBQuery, **kwargs: Any) List[VectorDBQueryResult] [source]#
Searches for similar vectors in the storage based on the provided query.
- Parameters:
query (VectorDBQuery) – The query object containing the search vector and the number of top similar vectors to retrieve.
**kwargs (Any) – Additional keyword arguments passed to search.
- Returns:
- A list of vectors retrieved from the
storage based on similarity to the query vector.
- Return type:
List[VectorDBQueryResult]
- status() VectorDBStatus [source]#
Retrieves the current status of the Milvus collection. This method provides information about the collection, including its vector dimensionality and the total number of vectors stored.
- Returns:
- An object containing information about the
collection’s status.
- Return type:
- class camel.storages.vectordb_storages.QdrantStorage(vector_dim: int, collection_name: str | None = None, url_and_api_key: Tuple[str, str] | None = None, path: str | None = None, distance: VectorDistance = VectorDistance.COSINE, delete_collection_on_del: bool = False, **kwargs: Any)[source]#
Bases:
BaseVectorStorage
An implementation of the BaseVectorStorage for interacting with Qdrant, a vector search engine.
The detailed information about Qdrant is available at: Qdrant
- Parameters:
vector_dim (int) – The dimenstion of storing vectors.
collection_name (Optional[str], optional) – Name for the collection in the Qdrant. If not provided, set it to the current time with iso format. (default:
None
)url_and_api_key (Optional[Tuple[str, str]], optional) – Tuple containing the URL and API key for connecting to a remote Qdrant instance. (default:
None
)path (Optional[str], optional) – Path to a directory for initializing a local Qdrant client. (default:
None
)distance (VectorDistance, optional) – The distance metric for vector comparison (default:
VectorDistance.COSINE
)delete_collection_on_del (bool, optional) – Flag to determine if the collection should be deleted upon object destruction. (default:
False
)**kwargs (Any) – Additional keyword arguments for initializing QdrantClient.
Notes
If url_and_api_key is provided, it takes priority and the client will attempt to connect to the remote Qdrant instance using the URL endpoint.
If url_and_api_key is not provided and path is given, the client will use the local path to initialize Qdrant.
If neither url_and_api_key nor path is provided, the client will be initialized with an in-memory storage (“:memory:”).
- add(records: List[VectorRecord], **kwargs) None [source]#
Adds a list of vectors to the specified collection.
- Parameters:
vectors (List[VectorRecord]) – List of vectors to be added.
**kwargs (Any) – Additional keyword arguments.
- Raises:
RuntimeError – If there was an error in the addition process.
- property client: Any#
Provides access to the underlying vector database client.
- delete(ids: List[str], **kwargs: Any) None [source]#
Deletes a list of vectors identified by their IDs from the storage.
- Parameters:
ids (List[str]) – List of unique identifiers for the vectors to be deleted.
**kwargs (Any) – Additional keyword arguments.
- Raises:
RuntimeError – If there is an error during the deletion process.
- query(query: VectorDBQuery, **kwargs: Any) List[VectorDBQueryResult] [source]#
Searches for similar vectors in the storage based on the provided query.
- Parameters:
query (VectorDBQuery) – The query object containing the search vector and the number of top similar vectors to retrieve.
**kwargs (Any) – Additional keyword arguments.
- Returns:
- A list of vectors retrieved from the
storage based on similarity to the query vector.
- Return type:
List[VectorDBQueryResult]
- status() VectorDBStatus [source]#
Returns status of the vector database.
- Returns:
The vector database status.
- Return type:
- class camel.storages.vectordb_storages.VectorDBQuery(query_vector: List[float], top_k: int)[source]#
Bases:
BaseModel
Represents a query to a vector database.
- query_vector#
The numerical representation of the query vector.
- Type:
List[float]
- top_k#
The number of top similar vectors to retrieve from the database. (default:
1
)- Type:
int, optional
- model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#
A dictionary of computed field names and their corresponding ComputedFieldInfo objects.
- model_config: ClassVar[ConfigDict] = {}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_fields: ClassVar[Dict[str, FieldInfo]] = {'query_vector': FieldInfo(annotation=List[float], required=True), 'top_k': FieldInfo(annotation=int, required=False, default=1)}#
Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.
This replaces Model.__fields__ from Pydantic V1.
- query_vector: List[float]#
The numerical representation of the query vector.
- top_k: int#
The number of top similar vectors to retrieve from the database.
- class camel.storages.vectordb_storages.VectorDBQueryResult(*, record: VectorRecord, similarity: float)[source]#
Bases:
BaseModel
Encapsulates the result of a query against a vector database.
- record#
The target vector record.
- Type:
- similarity#
The similarity score between the query vector and the record.
- Type:
float
- classmethod create(similarity: float, vector: List[float], id: str, payload: Dict[str, Any] | None = None) VectorDBQueryResult [source]#
A class method to construct a VectorDBQueryResult instance.
- model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#
A dictionary of computed field names and their corresponding ComputedFieldInfo objects.
- model_config: ClassVar[ConfigDict] = {}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_fields: ClassVar[Dict[str, FieldInfo]] = {'record': FieldInfo(annotation=VectorRecord, required=True), 'similarity': FieldInfo(annotation=float, required=True)}#
Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.
This replaces Model.__fields__ from Pydantic V1.
- record: VectorRecord#
- similarity: float#
- class camel.storages.vectordb_storages.VectorDBStatus(*, vector_dim: int, vector_count: int)[source]#
Bases:
BaseModel
Vector database status.
- vector_dim#
The dimention of stored vectors.
- Type:
int
- vector_count#
The number of stored vectors.
- Type:
int
- model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#
A dictionary of computed field names and their corresponding ComputedFieldInfo objects.
- model_config: ClassVar[ConfigDict] = {}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_fields: ClassVar[Dict[str, FieldInfo]] = {'vector_count': FieldInfo(annotation=int, required=True), 'vector_dim': FieldInfo(annotation=int, required=True)}#
Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.
This replaces Model.__fields__ from Pydantic V1.
- vector_count: int#
- vector_dim: int#
- class camel.storages.vectordb_storages.VectorRecord(*, vector: List[float], id: str = None, payload: Dict[str, Any] | None = None)[source]#
Bases:
BaseModel
Encapsulates information about a vector’s unique identifier and its payload, which is primarily used as a data transfer object when saving to vector storage.
- vector#
The numerical representation of the vector.
- Type:
List[float]
- id#
A unique identifier for the vector. If not provided, an random uuid will be assigned.
- Type:
str, optional
- payload#
Any additional metadata or information related to the vector. (default:
None
)- Type:
Optional[Dict[str, Any]], optional
- id: str#
- model_computed_fields: ClassVar[Dict[str, ComputedFieldInfo]] = {}#
A dictionary of computed field names and their corresponding ComputedFieldInfo objects.
- model_config: ClassVar[ConfigDict] = {}#
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- model_fields: ClassVar[Dict[str, FieldInfo]] = {'id': FieldInfo(annotation=str, required=False, default_factory=<lambda>), 'payload': FieldInfo(annotation=Union[Dict[str, Any], NoneType], required=False, default=None), 'vector': FieldInfo(annotation=List[float], required=True)}#
Metadata about the fields defined on the model, mapping of field names to [FieldInfo][pydantic.fields.FieldInfo] objects.
This replaces Model.__fields__ from Pydantic V1.
- payload: Dict[str, Any] | None#
- vector: List[float]#