Feast Feature Store

Overview

Feast is an open-source feature store developed to streamline the management and serving of machine learning features across diverse environments. Initially developed at Gojek, Feast addresses the operational challenges associated with providing consistent and low-latency access to features for both model training and real-time inference (Feast Documentation). It operates by abstracting the underlying data infrastructure, allowing data scientists and ML engineers to define features once and reuse them across multiple models and applications.

The system comprises several core components: Feast Core, which handles feature definition and metadata management; the Feast Offline Store, used for historical feature retrieval during model training and batch inference; and the Feast Online Store, which provides low-latency access to the latest feature values for real-time predictions. This separation ensures that the same feature logic is applied consistently, mitigating training-serving skew, a common issue in ML deployments (O'Reilly Radar on ML Skew).

Feast is designed for organizations that require a scalable and flexible solution for feature management, particularly those with complex data pipelines and multiple ML models in production. Its Python SDK enables declarative feature definition, integrating with popular data sources and serving layers. By centralizing feature definitions, Feast supports collaboration among teams, reduces redundant feature engineering efforts, and accelerates the iteration cycle for machine learning projects.

The platform's architecture supports various data sources, including Apache Kafka for streaming data, and integrates with data warehouses like Snowflake and Google BigQuery for offline feature retrieval. For online serving, it can connect to key-value stores such as Redis and Amazon DynamoDB. This versatility allows Feast to fit into existing ML infrastructure landscapes without requiring a complete overhaul.

Organizations choose Feast to achieve greater operational efficiency in their MLOps workflows. By providing a single source of truth for features, it helps enforce data governance, improve feature discoverability, and ensure that models are always trained and served with accurate, up-to-date data. Its open-source nature provides transparency and allows for community contributions and customization.

Key features

• Declarative Feature Definition: Define features using a Python SDK, specifying data sources, transformations, and serving parameters.
• Online and Offline Consistency: Ensures that features used for model training (offline) are identical to those used for real-time inference (online), preventing training-serving skew.
• Feature Versioning: Manages different versions of features, allowing for experimentation and rollback capabilities.
• Point-in-Time Correctness: Retrieves historical feature values accurately for model training, aligning features with event timestamps.
• Scalable Serving: Supports high-throughput, low-latency feature serving for real-time prediction through integration with online data stores.
• Data Source Connectors: Built-in support for various data sources, including data warehouses (e.g., BigQuery, Snowflake) and streaming platforms (e.g., Kafka).
• Feature Discoverability: Centralized feature definitions and metadata enable data scientists to discover and reuse existing features.
• Extensibility: Designed with an extensible architecture to allow integration with custom data sources and online/offline stores.

Pricing

Feast is open-source software and is available for use under the Apache License 2.0. There are no direct licensing costs associated with using Feast itself.

Users are responsible for the operational costs associated with the underlying infrastructure components (e.g., cloud computing, storage, databases, and managed services) that Feast integrates with (Feast Documentation).

Common integrations

• Data Warehouses: Integrates with cloud data warehouses like Google BigQuery and Snowflake for offline feature materialization.
• NoSQL Databases: Connects to online stores such as Amazon DynamoDB and Redis for low-latency feature serving.
• Stream Processing: Supports ingestion from real-time data streams like Apache Kafka.
• ML Frameworks: Designed to work with popular machine learning frameworks like TensorFlow, PyTorch, and scikit-learn.
• Orchestration Tools: Can be integrated into MLOps pipelines orchestrated by tools such as Apache Airflow.

Alternatives

• Tecton: An enterprise feature platform offering managed services, advanced governance, and integration with cloud ecosystems.
• Hopsworks: A platform that includes a managed feature store, ML orchestration, and data governance, available both in the cloud and on-premises.
• Amazon SageMaker Feature Store: A fully managed feature store service within AWS SageMaker, designed for native integration with AWS ML ecosystem.

Getting started

To get started with Feast, you typically define your feature repository, specify data sources, and then materialize and serve features. This example demonstrates defining a simple feature view and retrieving features using the Python SDK.

import pandas as pd
from feast import FeatureView, FeatureService, Field, ValueType
from feast.infra.offline_stores.file_source import FileSource

# Define an entity
# In a real scenario, this might come from a database or a stream
entity_df = pd.DataFrame(
    {
        "driver_id": [1001, 1002, 1003, 1004],
        "event_timestamp": [pd.Timestamp.now() for _ in range(4)],
    }
)

# Define a file source for the offline store
driver_hourly_stats = FileSource(
    path="data/driver_hourly_stats.parquet", # This would be a real data file
    timestamp_field="event_timestamp",
    created_timestamp_column="created_timestamp",
)

# Define a FeatureView
driver_hourly_stats_view = FeatureView(
    name="driver_hourly_stats",
    entities=["driver_id"],
    ttl=pd.Timedelta(days=1), # Time-to-live for features in the online store
    schema=[
        Field(name="conv_rate", dtype=ValueType.FLOAT),
        Field(name="acc_rate", dtype=ValueType.FLOAT),
    ],
    source=driver_hourly_stats,
)

# Define a FeatureService to group features for convenient retrieval
driver_feature_service = FeatureService(
    name="driver_feature_service",
    features=[driver_hourly_stats_view]
)

# To apply this definition, you would typically run 'feast apply' from your terminal
# After applying, you can retrieve features:
# from feast import FeatureStore
# store = FeatureStore(repo_path=".") # Path to your feature repository
# feature_vector = store.get_historical_features(
#     entity_df=entity_df,
#     feature_service=driver_feature_service
# ).to_df()
# print(feature_vector)

This Python code snippet outlines the foundational steps: defining a feature view and grouping features into a service. The actual data referenced by FileSource would typically be prepared and stored in a designated location. Running feast apply in the project directory would register these definitions with your Feast instance, making the features available for retrieval. Further details on local setup and deployment can be found in the Feast Getting Started guide.