Adapters

Adapters are lightweight, parameter-efficient fine-tuning components that enable you to customize language models without modifying the entire model weights. Factory automates the creation, training, and management of adapters, making it easy to fine-tune models for specific tasks.

What is an Adapter?

An adapter in Factory is a trained component that:

1. Modifies only a small subset of model parameters (typically less than 1%)
2. Preserves the base model's general capabilities
3. Adds domain-specific knowledge or task abilities
4. Requires significantly less compute than full fine-tuning

Factory implements adapters using the LoRA (Low-Rank Adaptation) technique, which adds small trainable matrices to specific layers of the model.

Creating and Training an Adapter

To create and train an adapter in Factory:

from factory_sdk import FactoryClient, TrainArgs, AdapterArgs, InitArgs
 
# Initialize the Factory client
factory = FactoryClient(
    tenant="your_tenant_name",
    project="your_project_name",
    token="your_api_key",
)
 
# Create and train the adapter
adapter = factory.adapter \
    .with_name("financial-phrases") \
    .based_on_recipe(recipe) \
    .using_model(model) \
    .with_hyperparameters(
        TrainArgs(
            train_batch_size=8,
            eval_batch_size=8,
            gradient_accumulation_steps=2,
            num_train_epochs=2,
            eval_every_n_minutes=2,
            max_eval_samples=100
        ),
        AdapterArgs(
            layer_selection_percentage=.5,
            rank="auto",
            alpha="auto"
        ),
        InitArgs(
            n_test_samples=200
        )
    ) \
    .run()

When you call .run(), Factory:

1. Downloads the model and dataset locally
2. Performs layer selection to determine which layers to fine-tune
3. Estimates the optimal rank for LoRA adapters
4. Trains the adapter with the specified hyperparameters
5. Periodically evaluates and uploads checkpoints
6. Tracks metrics and performance throughout training

Advanced Adapter Configuration

Training Arguments (TrainArgs)

These arguments control the overall training process:

Adapter Arguments (AdapterArgs)

These arguments control the adapter architecture:

Initialization Arguments (InitArgs)

These arguments control the initialization phase:

Automatic Parameter Optimization

Factory includes intelligent systems to optimize adapter training:

Automatic Layer Selection

Setting layer_selection_percentage (e.g., to 0.5) enables Factory to:

1. Analyze model layers to determine their importance for your specific task
2. Select the most sensitive layers that have the greatest impact on performance
3. Only train adapters for these selected layers, improving efficiency

Automatic Rank Determination

When you set rank="auto", Factory:

1. Analyzes activation patterns in selected layers
2. Computes the optimal rank based on effective dimensionality
3. Sets different ranks for different layers based on their complexity
4. Balances performance vs. parameter count

Adapter Versioning

Like models and datasets, adapters are versioned in Factory:

1. AdapterMeta: The main reference for an adapter under a specific name
2. AdapterRevision: A specific version of an adapter with trained weights

When you update an adapter by training again with the same name, Factory creates a new revision while maintaining the connection to previous versions.

Monitoring Training Progress

During training, Factory automatically:

1. Tracks training loss, learning rate, and evaluation metrics
2. Monitors GPU utilization and performance statistics
3. Creates checkpoints at regular intervals
4. Uploads metrics and checkpoints to the Factory Hub

You can monitor all these metrics in real-time in the Factory Hub.

Example Workflow

Here's a complete example of creating, training, and using an adapter:

from factory_sdk import FactoryClient, TrainArgs, AdapterArgs, InitArgs
 
# Initialize client
factory = FactoryClient(
    tenant="your_tenant_name",
    project="your_project_name",
    token="your_api_key",
)
 
# Load base model
model = factory.base_model.with_name("qwen_small") \
    .from_open_weights("Qwen/Qwen2.5-0.5B-Instruct") \
    .save_or_fetch()
 
# Load and prepare dataset
data = load_dataset("takala/financial_phrasebank", "sentences_allagree")
data = data["train"].train_test_split(test_size=0.1, seed=42)
dataset = factory.dataset.with_name("financial-phrases") \
    .from_local(data) \
    .save_or_fetch()
 
# Create recipe
recipe = factory.recipe \
    .with_name("financial-phrases") \
    .using_dataset(dataset) \
    .with_preprocessor(your_processor_function) \
    .save_or_fetch()
 
# Create and train adapter
adapter = factory.adapter \
    .with_name("financial-phrases") \
    .based_on_recipe(recipe) \
    .using_model(model) \
    .with_hyperparameters(
        TrainArgs(train_batch_size=8, num_train_epochs=3),
        AdapterArgs(layer_selection_percentage=0.5),
        InitArgs(n_test_samples=500)
    ) \
    .run()
 
# Deploy the adapter
deployment = factory.deployment \
    .with_name("financial-sentiment") \
    .for_adapter(adapter) \
    .with_config(DeploymentArgs(port=8000)) \
    .run()

Best Practices for Adapter Training

• Start small: Begin with a small number of epochs (1-3) to assess performance
• Use automatic optimization: Let Factory select layers and ranks with "auto" settings
• Enable quantization: 4-bit quantization (the default) dramatically reduces memory usage
• Monitor evaluation metrics: Check the Factory Hub to see when performance plateaus
• Experiment with layer percentages: Try different values of layer_selection_percentage (0.3-0.7)
• Adjust batch size: If you encounter out-of-memory errors, reduce batch size and increase gradient accumulation