Train Models

Factory provides a streamlined approach to fine-tuning small language models (LLMs) with minimal compute resources while achieving production-quality results. This guide explains how to train models efficiently using the Factory SDK.

Training Overview

The training process in Factory consists of four main components:

1. Base Models - Pre-trained foundation models that serve as your starting point
2. Datasets - Structured data containing examples for your specific task
3. Recipes - Transformations that convert raw data into training examples
4. Adapters - Lightweight components that modify only a fraction of model parameters

By focusing on parameter-efficient fine-tuning through adapters, Factory enables you to customize models for specific domains without the computational expense of full fine-tuning.

Key Training Features

Factory's training approach offers several advantages:

• Parameter-Efficient Fine-Tuning - Update less than 1% of model parameters
• Automatic Layer Selection - Train only the most impactful model layers
• Optimal Rank Determination - Automatically find the ideal adapter dimensions
• Memory-Efficient Training - Use 4-bit quantization to reduce memory requirements

Training Workflow

graph LR
    A[Load Base Model] --> B[Prepare Dataset]
    B --> C[Create Recipe]
    C --> D[Train Adapter]
    D --> E[Evaluate Results]
    E --> F[Deploy Model]

The typical training workflow involves:

1. Loading a base model from Hugging Face or another source
2. Preparing a dataset with training and test splits
3. Creating a recipe to transform raw data into training examples
4. Training an adapter with parameter-efficient techniques
5. Evaluating results and comparing performance
6. Deploying the fine-tuned model for inference

Getting Started

To start training your first model with Factory:

from factory_sdk import FactoryClient
 
# Initialize the Factory client
factory = FactoryClient(
    tenant="your_tenant_name",
    project="your_project_name",
    token="your_api_key",
)
 
# Load a base model
model = factory.base_model.with_name("qwen_small") \
    .from_open_weights("Qwen/Qwen2.5-0.5B-Instruct") \
    .save_or_fetch()
 
# Train an adapter
adapter = factory.adapter \
    .with_name("my-first-adapter") \
    .based_on_recipe(recipe) \
    .using_model(model) \
    .with_hyperparameters(...) \
    .run()

Training Components