OpenAI Assistants with persistent vector store

query_builder 20 min

The Astra Assistants API is a drop in replacement for OpenAI’s Assistants API that uses Serverless (Vector) database for persistence and supports the following features:

Full compatibility with the OpenAI Assistants API v2, including messages, assistants, threads, runs, vector_stores, and files.
Third-party embeddings and completion models with hundreds of LLMs, including Anthropic, Gemini, Mistral, Groq, LLama, and Cohere, powered by liteLLM.
Ollama support for local models.
Open source with options for managed service or self hosting.
Function calling and file search.
Data privacy and protection.

The database stores and queries embeddings for retrieval augmented generation (RAG). For large language model (LLM) tasks, such as embedding generation and chat completion, the database calls OpenAI or other LLMs.

Users interact with the service through the OpenAI SDKs. Store your proprietary data and run assistant API examples on your own Astra DB Serverless database, which can be managed, accessed, and secured.

Prerequisites

To complete this tutorial, you’ll need the following:

An active Astra account
A paid OpenAI account
Python 3.10 or later
An application token with the Database Administrator role

You should also be proficient in the following tasks:

Running a basic Python script.

Run an Assistant API example

Create a .env file with the environment variables for your selected model:

OpenAI
Perplexity
Cohere
Bedrock
Vertex

.env

#!/bin/bash

# Go to https://astra.datastax.com > "Tokens" to generate an Administrator User token.
export ASTRA_DB_APPLICATION_TOKEN=
# Go to https://platform.openai.com/api-keys to create a secret key.
export OPENAI_API_KEY=

.env

#!/bin/bash

# Go to https://astra.datastax.com > "Tokens" to generate an Administrator User token.
export ASTRA_DB_APPLICATION_TOKEN=
# Go to https://platform.openai.com/api-keys to create a secret key.
export OPENAI_API_KEY=

# Go to https://www.perplexity.ai/settings/api to generate a secret key.
export PERPLEXITYAI_API_KEY=

.env

#!/bin/bash

# Go to https://astra.datastax.com > "Tokens" to generate an Administrator User token.
export ASTRA_DB_APPLICATION_TOKEN=
# Go to https://platform.openai.com/api-keys to create a secret key.
export OPENAI_API_KEY=

# Go to https://dashboard.cohere.com/api-keys to create an API key.
export COHERE_API_KEY=

.env

#!/bin/bash

# Go to https://astra.datastax.com > Tokens to generate an Administrator User token.
export ASTRA_DB_APPLICATION_TOKEN=
# Go to https://platform.openai.com/api-keys to create a secret key.
export OPENAI_API_KEY=

# Bedrock models: https://docs.aws.amazon.com/bedrock/latest/userguide/setting-up.html
export AWS_REGION_NAME=
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=

.env

#!/bin/bash

# Go to https://astra.datastax.com > Tokens to generate an Administrator User token.
export ASTRA_DB_APPLICATION_TOKEN=
# Go to https://platform.openai.com/api-keys to create a secret key.
export OPENAI_API_KEY=

# Vertex AI models: https://console.cloud.google.com/vertex-ai
export GOOGLE_JSON_PATH=
export GOOGLE_PROJECT_ID=

Install poetry:

curl -sSL https://install.python-poetry.org | python3 -

Install the dependencies:

poetry install astra-assistants openai python-dotenv

Build the Assistants API-powered application

Import and patch your client:

from openai import OpenAI
from astra_assistants import patch
client = patch(OpenAI())

Using your token, the system creates an Astra DB Serverless database named assistant_api_db. The first request can take a few minutes to create your database.

Create your assistant:

assistant = client.beta.assistants.create(
  instructions="You are a personal math tutor. When asked a math question, write and run code to answer the question.",
  model="gpt-4o",
)

By default, the service uses Astra DB Serverless as the vector store and OpenAI for embeddings and chat completion.

Third-party LLM support

Astra DB supports many third-party models for embeddings and completion with litellm.

You must pass your service’s API key using api-key and embedding-model headers.

You can pass different models with the corresponding API key in your environment:

model="gpt-4o"

assistant = client.beta.assistants.create(
    name="Math Tutor",
    instructions="You are a personal math tutor. Answer questions briefly, in a sentence or less.",
    model=model,
)

model="openai/gpt-4o-mini"

assistant = client.beta.assistants.create(
    name="Math Tutor",
    instructions="You are a personal math tutor. Answer questions briefly, in a sentence or less.",
    model=model,
)

model="cohere/command-r-plus"

assistant = client.beta.assistants.create(
    name="Math Tutor",
    instructions="You are a personal math tutor. Answer questions briefly, in a sentence or less.",
    model=model,
)

model="perplexity/mixtral-8x7b-instruct"

assistant = client.beta.assistants.create(
    name="Math Tutor",
    instructions="You are a personal math tutor. Answer questions briefly, in a sentence or less.",
    model=model,
)

model="perplexity/pplx-70b-online"

assistant = client.beta.assistants.create(
    name="Math Tutor",
    instructions="You are a personal math tutor. Answer questions briefly, in a sentence or less.",
    model=model,
)

model="anthropic/claude-3-5-sonnet"

assistant = client.beta.assistants.create(
    name="Math Tutor",
    instructions="You are a personal math tutor. Answer questions briefly, in a sentence or less.",
    model=model,
)

model="gemini/gemini-1.5-flash"

assistant = client.beta.assistants.create(
    name="Math Tutor",
    instructions="You are a personal math tutor. Answer questions briefly, in a sentence or less.",
    model=model,
)

For third-party embedding models, DataStax supports the embedding_model in client.files.create:

file = client.files.create(
    file=open(
        "./test/language_models_are_unsupervised_multitask_learners.pdf",
        "rb",
    ),
    purpose="assistants",
    embedding_model="text-embedding-3-large",
)

By default, the API uses your Astra DB Serverless database as the vector store and OpenAI for the embeddings and chat completion.

OpenAI Assistants with persistent vector store

Prerequisites

Run an Assistant API example

Build the Assistants API-powered application

Third-party LLM support

See also

Was this helpful?

Give Feedback