Graphcore

Overview

Graphcore, founded in 2016, specializes in the development of hardware and software solutions specifically engineered for artificial intelligence workloads. The company's core offering is the Intelligence Processing Unit (IPU), a processor architecture designed to accelerate machine learning training and inference tasks. The IPU architecture differs from general-purpose CPUs and GPUs by incorporating a massively parallel, fine-grained processing structure with on-chip memory close to each core, aiming to reduce memory latency and enhance data throughput for AI models Graphcore IPU technology overview.

Graphcore's IPU systems are intended for organizations engaged in computationally intensive AI research, development, and deployment. This includes enterprises training large foundational models, academic institutions conducting cutting-edge AI research, and developers building high-performance inference systems for real-time applications. The company positions its technology as an alternative to traditional GPU-based systems, emphasizing its suitability for specific graph-native computations prevalent in many AI algorithms.

The IPU system comprises both the hardware accelerators and a software stack, the Poplar SDK. This SDK provides a comprehensive set of tools, libraries, and frameworks that enable developers to port, optimize, and deploy AI models on IPU hardware. Supported frameworks include TensorFlow, PyTorch, and PaddlePaddle. Graphcore's approach to AI processing aims to provide scalability and efficiency across various deployment scenarios, from cloud-based data centers to edge computing environments, by offering different IPU form factors like PCIe cards and integrated systems.

Graphcore's technology is often evaluated for its performance characteristics in specific AI benchmarks, particularly those involving graph neural networks and other sparse or irregular computational patterns where its memory architecture may offer advantages. Industry analysis often compares the efficiency and scalability of IPUs against established GPU architectures for various AI workloads Gartner's AI Accelerator Market Guide. The company targets use cases such as natural language processing, computer vision, and scientific computing, where large models demand significant computational resources.

Key features

• Intelligence Processing Units (IPUs): Specialized processors designed for AI and machine learning workloads, featuring a many-core architecture with large on-chip memory.
• Poplar SDK: A comprehensive software development kit for IPUs, including compilers, PopLibs (optimized libraries), graph tools, and interfaces for popular AI frameworks like TensorFlow and PyTorch.
• High Bandwidth Memory (HBM): Integration of high-bandwidth memory for rapid data access, aiming to reduce bottlenecks common in large AI models.
• IPU-Fabric connectivity: Interconnect technology designed for scaling IPU systems, enabling multiple IPUs to work cooperatively on single models or multiple independent tasks.
• Graph-native architecture: Optimized for graph-based computations, which are common in many AI algorithms, potentially offering efficiency gains for specific model types.
• Cloud and On-Premise Deployment: Offers IPU systems for deployment in data centers, cloud environments, and specialized edge applications.
• Sparse Model Optimization: Features designed to efficiently handle sparse matrix operations and irregular data patterns, relevant for certain deep learning architectures.

Pricing

Graphcore offers custom enterprise pricing for its IPU hardware and software solutions. Specific costs are determined based on the scale of deployment, the configuration of IPU systems required, and the associated support and service agreements. Prospective customers typically engage directly with Graphcore's sales team for tailored quotes.

For detailed pricing information and current offerings, direct consultation with Graphcore sales is necessary.

Common integrations

• TensorFlow: Graphcore's Poplar SDK provides direct integration and optimized support for TensorFlow models running on IPU hardware Graphcore TensorFlow integration documentation.
• PyTorch: The PopTorch library allows PyTorch models to be compiled and executed efficiently on Graphcore IPUs, abstracting hardware specifics Graphcore PyTorch integration guide.
• Hugging Face Transformers: Compatibility with models from the Hugging Face Transformers library enables deployment of various NLP models on IPUs Hugging Face Graphcore documentation.
• PaddlePaddle: Support for Baidu's deep learning framework PaddlePaddle on IPU systems, expanding the range of compatible AI applications.
• ONNX Runtime: Integration with ONNX Runtime for optimized inference of models in the Open Neural Network Exchange format.
• Kubernetes: IPU systems can be managed within Kubernetes clusters for orchestration and scaling of AI workloads.

Alternatives

• NVIDIA: Offers a wide range of GPUs (e.g., A100, H100) and CUDA software stack, widely used for AI training and inference.
• Intel: Provides AI accelerators like Intel Gaudi (Habana Labs) and integrated AI capabilities in Xeon CPUs, along with development tools like OpenVINO.
• Cerebras Systems: Develops the Wafer-Scale Engine (WSE), aimed at accelerating large AI model training with a single, massive chip.
• AWS Inferentia/Trainium: Custom-designed AI chips by Amazon Web Services for cloud-based inference and training workloads.
• Google TPUs: Tensor Processing Units developed by Google specifically for accelerating deep learning workloads within Google Cloud.

Getting started

Getting started with Graphcore IPUs typically involves setting up the Poplar SDK and then running a model. The following example demonstrates a basic PyTorch model configured to run on an IPU using PopTorch.

import torch
import torch.nn as nn
import poptorch

# 1. Define a simple PyTorch model
class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(10, 1)

    def forward(self, x):
        return self.linear(x)

# 2. Instantiate the model
model = SimpleModel()

# 3. Create a PopTorch Options object (for IPU configuration)
# This is where you can specify IPU-specific settings
opts = poptorch.Options()
opts.deviceIterations(1) # Number of iterations on the IPU per host iteration
opts.replicationFactor(1) # Number of IPUs to replicate the model across
opts.outputMode(poptorch.OutputMode.ALL) # Control what is returned from the IPU

# 4. Wrap the model with PopTorch for IPU execution
iput_model = poptorch.trainingModel(model, options=opts)

# 5. Prepare dummy input data
input_data = torch.randn(64, 10)
target_data = torch.randn(64, 1)

# 6. Define a loss function and optimizer
loss_fn = nn.MSELoss()
optimizer = poptorch.optim.SGD(iput_model.parameters(), lr=0.01)

# 7. Training loop (simplified)
print("Starting training on IPU...")
for epoch in range(5):
    # Forward pass and backward pass on IPU
    output = iput_model(input_data)
    loss = loss_fn(output, target_data)

    # PopTorch handles gradient accumulation and weight updates on IPU
    # within the iput_model call or with explicit optimizer.step()
    # For simplicity, we'll simulate an update here. In a real scenario,
    # the iput_model call handles the step if using poptorch.trainingModel.

    # In a typical PopTorch training loop, you might do:
    # output, loss = iput_model(input_data, target_data)
    # loss.item() to get the scalar loss

    # For this simplified example, let's just print the loss
    print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")

print("Training complete.")

# 8. Detach and cleanup (important for resource management)
iput_model.destroy()

This example initializes a simple neural network, configures PopTorch options for IPU execution, wraps the PyTorch model, and demonstrates a basic training loop. In a production environment, this would involve more complex data loading, model architectures, and thorough optimization for IPU performance.