Weaviate

Overview

Weaviate is an open-source, low-latency vector database designed to store objects and their vector embeddings, enabling semantic search and various AI-powered applications. Founded in 2019, it functions as a cloud-native, real-time database that can scale horizontally to manage billions of data objects. The core utility of Weaviate lies in its ability to convert unstructured data (text, images, audio, video) into vectors using machine learning models, then store and index these vectors for fast retrieval based on semantic similarity rather than exact keyword matches.

Developers and technical buyers utilize Weaviate primarily for applications requiring understanding of data meaning. This includes building semantic search engines that can find relevant results even if exact keywords are not present. Another significant use case is Retrieval Augmented Generation (RAG), where Weaviate retrieves contextually relevant information to augment large language models (LLMs), improving the accuracy and relevance of their generated responses. Furthermore, it supports the creation of recommendation engines by identifying similar items or users based on their vector representations, and enables real-time data analysis for pattern recognition and anomaly detection across large datasets.

Weaviate offers two main deployment options: Weaviate Cloud (WCD), a managed service, and an open-source version for self-hosting. The managed cloud service provides a free sandbox environment for initial exploration, with paid tiers designed for production workloads. The open-source version allows for deployment on various infrastructures, giving users full control over their data and environment. Weaviate's architecture supports a flexible, graph-like data schema, which can be extended and modified as application requirements evolve. This adaptability, combined with its focus on vector indexing and search, positions Weaviate as a foundational component for AI-centric data infrastructure.

Key features

• Vector Storage and Indexing: Stores data objects alongside their vector embeddings, using efficient indexing algorithms for fast similarity search.
• Semantic Search: Enables searching for data based on its meaning rather than just keyword matching, powered by vector similarity.
• Retrieval Augmented Generation (RAG): Facilitates the retrieval of relevant external knowledge to enhance the context for large language models.
• Recommendation Engines: Supports building systems that suggest items or content based on similarity to user preferences or other items.
• Real-time Data Analysis: Processes and queries vector data in real time, enabling dynamic insights and applications.
• Modular Vectorization: Integrates with various machine learning models and APIs (e.g., OpenAI, Hugging Face, Cohere) for generating embeddings, allowing flexibility in model choice.
• Graph-like Data Schema: Provides a flexible data model that allows defining classes, properties, and cross-references between data objects.
• Scalability: Designed for horizontal scalability to handle large datasets and high query loads, suitable for enterprise applications.
• Open Source & Cloud Offering: Available as a self-hostable open-source project and a managed service via Weaviate Cloud (WCD).
• Multi-tenancy: Weaviate Cloud supports multi-tenancy, allowing isolation of data for different projects or clients within a single cluster.
• Data Compliance: Adheres to compliance standards such as SOC 2 Type II and GDPR, important for regulated industries.

Pricing

Weaviate offers a free sandbox for its Weaviate Cloud service, alongside an open-source version that can be self-hosted without licensing costs. Paid tiers for Weaviate Cloud are structured based on resource consumption.

Detailed pricing information, including specific resource allocations for each tier, is available on the Weaviate pricing page. The Scale plan's cost increases with additional resource usage, such as data storage, queries per second, and vector dimensions.

Common integrations

• OpenAI: Integrate with OpenAI's embedding and generative models for vectorization and RAG.
• Hugging Face: Utilize Hugging Face models for text embedding and other NLP tasks directly within Weaviate.
• Google Cloud Vertex AI: Connect with Google's Vertex AI for accessing a range of machine learning services, including embeddings.
• AWS: Deploy Weaviate on Amazon Web Services (AWS) infrastructure, leveraging EC2, S3, and other services.
• Apache Kafka: Integrate with Kafka for real-time data ingestion and stream processing into Weaviate.
• LangChain: Use Weaviate as a vector store within LangChain applications for building LLM-powered agents and RAG pipelines.
• LlamaIndex: Integrate with LlamaIndex to enhance data indexing and retrieval for LLM applications.

Alternatives

• Pinecone: A managed vector database service focusing on high-performance similarity search for AI applications.
• Qdrant: An open-source vector similarity search engine and database, available for self-hosting or as a managed cloud service.
• Milvus: An open-source vector database built for scalable similarity search and AI applications, with cloud-native design.
• Databricks Vector Search: A feature within the Databricks platform offering vector search capabilities for data stored in Delta Lake.
• Google Cloud Vertex AI Matching Engine: A managed service for large-scale nearest neighbor search, part of Google Cloud's AI platform.

Getting started

To begin using Weaviate, you can either deploy the open-source version locally or utilize the Weaviate Cloud Sandbox. The following Python example demonstrates connecting to a Weaviate instance, creating a schema, and adding data, then performing a semantic search. This example assumes you have a Weaviate instance running and accessible.

import weaviate
import json

# Best practice: Use environment variables for Weaviate URL and API key
WEAVIATE_URL = "YOUR_WEAVIATE_URL" # e.g., "http://localhost:8080" or your WCD URL
WEAVIATE_API_KEY = "YOUR_WEAVIATE_API_KEY" # Required for WCD, optional for local

# Connect to Weaviate
client = weaviate.Client(
    url=WEAVIATE_URL,
    auth_client_secret=weaviate.AuthApiKey(api_key=WEAVIATE_API_KEY) if WEAVIATE_API_KEY else None
)

# Check if the connection is successful
if client.is_ready():
    print("Successfully connected to Weaviate!")
else:
    print("Failed to connect to Weaviate. Check your URL and API key.")
    exit()

# Define a schema for a 'Article' class
article_class_schema = {
    "class": "Article",
    "description": "A collection of articles",
    "vectorizer": "text2vec-openai", # Or 'text2vec-huggingface' etc.
    "properties": [
        {
            "name": "title",
            "dataType": ["text"],
            "description": "The title of the article",
        },
        {
            "name": "content",
            "dataType": ["text"],
            "description": "The content of the article",
        },
        {
            "name": "url",
            "dataType": ["text"],
            "description": "The URL of the article",
        },
    ],
}

# Ensure the class doesn't exist already, then create it
if not client.schema.exists(article_class_schema["class"]):
    client.schema.create_class(article_class_schema)
    print(f"Created schema for class '{article_class_schema['class']}'")
else:
    print(f"Class '{article_class_schema['class']}' already exists. Skipping schema creation.")

# Add data objects
data_objects = [
    {
        "title": "The Future of AI in Healthcare",
        "content": "Artificial intelligence is transforming healthcare by improving diagnostics, personalizing treatment, and streamlining operations.",
        "url": "https://example.com/ai-healthcare"
    },
    {
        "title": "Advancements in Renewable Energy",
        "content": "New technologies are making solar and wind power more efficient and accessible, driving the global shift to sustainable energy.",
        "url": "https://example.com/renewable-energy"
    },
    {
        "title": "Impact of Quantum Computing",
        "content": "Quantum computing promises to revolutionize fields like cryptography, drug discovery, and materials science with unprecedented computational power.",
        "url": "https://example.com/quantum-computing"
    },
]

for obj in data_objects:
    client.data_object.create(
        data_object=obj,
        class_name="Article"
    )
print("Added data objects to Weaviate.")

# Perform a semantic search
search_query = "innovations in sustainable power"

response = (
    client.query
    .get("Article", ["title", "content", "url"])
    .with_near_text({"concepts": [search_query]})
    .with_limit(2)
    .do()
)

print(f"\nSearch results for '{search_query}':")
for result in response["data"]["Get"]["Article"]:
    print(f"  Title: {result['title']}")
    print(f"  Content: {result['content'][:100]}...")
    print(f"  URL: {result['url']}\n")

# Cleanup (optional): delete the class
# client.schema.delete_class("Article")
# print("Deleted 'Article' class.")

This example initializes a Weaviate client, defines a data schema for 'Article' objects, adds sample articles, and then executes a semantic search using with_near_text. The results returned are based on the conceptual similarity to the query, demonstrating Weaviate's core capability for vector-based retrieval. For more detailed instructions on installation and advanced features, refer to the Weaviate quickstart guide.