Migrate to a new embedding model for a collection (Python)

Follow this migration guide if you want to switch embedding models for your collection.

This migration only works for documents that have a $vectorize field.

Create a new collection with the desired embedding provider integration. For examples, see Create a collection that can automatically generate vector embeddings.

Migrate your documents to the new collection.

Exclude the $vector field from the migrated documents. The embedding provider integration for your new collection will automatically generate vector embeddings based on the $vectorize field and store them in the $vector field. Any documents without a $vectorize field will not have their $vector field automatically populated.

For example:

from astrapy import DataAPIClient

client = DataAPIClient()
database = client.get_database(
    "API_ENDPOINT", token="APPLICATION_TOKEN"
)

old_collection = database.get_collection("OLD_COLLECTION_NAME")
new_collection = database.get_collection("NEW_COLLECTION_NAME")

page_state = None
migrated_count = 0

# Use an empty filter to migrate all documents
filter = {}

# You must explicitly include $vectorize.
# $vector is excluded by default.
# _id and any other fields that don't start with $ are included by default.
projection = {"$vectorize": True}

while True:
    if page_state:
        cursor = old_collection.find(
            filter, projection=projection, initial_page_state=page_state
        )
    else:
        cursor = old_collection.find(filter, projection=projection)

    page = cursor.fetch_next_page()
    documents = page.results
    page_state = page.next_page_state

    if not documents:
        print("No more documents. Migration complete.")
        break

    # Insert the documents to the new collection.
    # _id and the other field values (excluding $vector) will be the same.
    # $vector will automatically be generated based on the value $vectorize.
    new_collection.insert_many(documents)

    migrated_count += len(documents)

    print(
        f"Migrated {migrated_count} documents. Page state: {page_state}"
    )

    if page_state is None:
        print("Reached final page. Migration complete.")
        break

Optionally, delete the collection that stores the old vector embeddings.

Migrate to a new embedding model for a collection (Python)

Was this helpful?

Give Feedback