Migrate to a new embedding model for a collection (TypeScript)

Follow this migration guide if you want to switch embedding models for your collection.

This migration only works for documents that have a $vectorize field.

Create a new collection with the desired embedding provider integration. For examples, see Create a collection that can automatically generate vector embeddings.

Migrate your documents to the new collection.

Exclude the $vector field from the migrated documents. The embedding provider integration for your new collection will automatically generate vector embeddings based on the $vectorize field and store them in the $vector field. Any documents without a $vectorize field will not have their $vector field automatically populated.

For example:

import {
  DataAPIClient,
  CollectionInsertManyError,
} from "@datastax/astra-db-ts";

const client = new DataAPIClient("APPLICATION_TOKEN");
const database = client.db("API_ENDPOINT");

const oldCollection = database.collection("OLD_COLLECTION_NAME");
const newCollection = database.collection("NEW_COLLECTION_NAME");

let pageState: string | null = null;
let migratedCount = 0;

// Use an empty filter to migrate all documents
const filter = {};

// You must explicitly include $vectorize.
// $vector is excluded by default.
// _id and any other fields that don't start with $ are included by default.
const projection = { $vectorize: true };

(async function () {
  while (true) {
    const cursor = oldCollection.find(filter, {
      projection,
      ...(pageState ? { initialPageState: pageState } : {}),
    });

    const page = await cursor.fetchNextPage();
    const documents = page.result;
    pageState = page.nextPageState;

    if (!documents.length) {
      console.log("No more documents. Migration complete.");
      break;
    }

    // Insert the documents to the new collection.
    // _id and the other field values (excluding $vector) will be the same.
    // $vector will automatically be generated based on the value of $vectorize.
    try {
      await newCollection.insertMany(documents);
    } catch (error) {
      if (error instanceof CollectionInsertManyError) {
        console.log(error.insertedIds());
      }
    }

    migratedCount += documents.length;

    console.log(
      `Migrated ${migratedCount} documents. Page state: ${pageState}`,
    );

    if (!pageState) {
      console.log("Reached final page. Migration complete.");
      break;
    }
  }
})();

Optionally, delete the collection that stores the old vector embeddings.

Migrate to a new embedding model for a collection (TypeScript)

Was this helpful?

Give Feedback