Cartridges

Cartridges are a prompt-learning method that stores a compressed long-context representation as a parameterized KV-cache prefix. The core idea comes from the paper Cartridges: Lightweight and general-purpose long context representations via self-study.

For a high-level overview and motivation, see the blog post Cartridges: Storing long contexts in tiny caches with self-study.

How Cartridges differ from Prefix Tuning

Both Prefix Tuning and Cartridges are served by injecting past_key_values (a prefix KV cache) into the base model.

Prefix Tuning learns virtual token embeddings (and optionally an MLP projection) and produces a KV prefix.
Cartridges learn the KV prefix itself directly (the per-layer key/value vectors for p virtual tokens), and are designed to be initialized from real prefill KV (for example, the first p tokens of a corpus/system prompt).

The paper also recommends freezing the first token as an attention sink for stability (num_frozen_tokens=1 is the default).

Usage (inference)

Load a trained CARTRIDGE adapter and run generation:

from transformers import AutoModelForCausalLM, AutoTokenizer

from peft import PeftModel

model_id = "Qwen/Qwen2.5-0.5B-Instruct"
adapter_path = "path/to/cartridge_adapter"

base = AutoModelForCausalLM.from_pretrained(model_id)
model = PeftModel.from_pretrained(base, adapter_path)

tok = AutoTokenizer.from_pretrained(model_id)
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

out = model.generate(**tok("Question about the corpus:", return_tensors="pt"), max_new_tokens=64)
print(tok.decode(out[0], skip_special_tokens=True))

If you need to create and initialize a cartridge before training, see the initialization options below.

Initialization options

The paper discusses a few practical initialization strategies:

Random KV (default): create a CartridgeConfig and start training. This initializes the KV prefix randomly.
KV from the first tokens of a prompt/corpus: use initialize_kv_prefix_from_text(model, tokenizer, text=...). This runs a prefill on text and copies the resulting KV cache for the first num_virtual_tokens into the adapter.
KV from an existing cache: use initialize_kv_prefix_from_past_key_values(model, past_key_values=...) if you already have a past_key_values object from a base-model prefill.

Training

The Cartridges paper proposes a SELF-STUDY distillation objective (a frozen base model provides teacher logits; the CARTRIDGE adapter is trained so the student matches the teacher’s next-token distribution over the target segment). PEFT keeps training logic out of the core library; see https://github.com/huggingface/peft/tree/main/examples/cartridge_self_study for a reference workflow. The example scripts use the frozen base model as the teacher and the adapted model as the student, so both share the same underlying checkpoint.

Composition

To concatenate independently trained cartridges into a single adapter, use compose_cartridge_adapters(...).

CartridgeConfig

class peft.CartridgeConfig

< source >

( task_type: Optional[Union[str, TaskType]] = None peft_type: Optional[Union[str, PeftType]] = None auto_mapping: Optional[dict] = None peft_version: Optional[str] = None base_model_name_or_path: Optional[str] = None revision: Optional[str] = None inference_mode: bool = False num_virtual_tokens: int = None token_dim: int = None num_transformer_submodules: Optional[int] = None num_attention_heads: Optional[int] = None num_layers: Optional[int] = None modules_to_save: Optional[list[str]] = None num_frozen_tokens: int = 1 )

Parameters

num_frozen_tokens (int, defaults to 1) — Number of prefix tokens at the start of the cartridge to keep frozen (no gradients). The Cartridges paper recommends freezing the first token as an attention sink for stability (set this to 1), as many LLMs use early tokens as attention sinks and changing them can harm training.

Configuration for CARTRIDGE, a KV-cache-parameterized prefix adapter.

This is similar to prefix-tuning in how it is served (as past_key_values), but it stores the KV cache directly as trainable parameters instead of learning it via an MLP projection.

Initialization: The Cartridges paper discusses multiple initialization options. In PEFT, initialization is a separate step from constructing the adapter config:

Random KV initialization (paper option 2): Create the adapter via get_peft_model(...). The CARTRIDGE prompt encoder parameters are randomly initialized by PyTorch.
KV derived from the first tokens of a prompt/corpus (paper option 3): Run a no-grad prefill on the base model and copy the first num_virtual_tokens cached KV tokens into the adapter. PEFT provides utilities for this (importable from peft or from peft.tuners.cartridge.utils):
- initialize_kv_prefix_from_text(model, tokenizer, text=...)
- initialize_kv_prefix_from_past_key_values(model, past_key_values=...)
If you already have a flattened KV-prefix tensor, you can load it directly via the prompt encoder’s load_prompt_embeddings(...) method.

CartridgeEncoder

class peft.CartridgeEncoder

< source >

( config )

A parameterized prefix KV cache.

The parameters are stored in the same flattened layout as PrefixEncoder output: `[num_virtual_tokens, num_layers

2 * token_dim], where token_dimis per-head hidden size times number of heads (after any GQA adjustment performed by_prepare_prompt_learning_config`).

If num_frozen_tokens > 0, the first num_frozen_tokens virtual tokens are stored as a non-trainable parameter, and the remaining tokens are trainable.

load_prompt_embeddings

< source >

( prompt_embeddings: torch.Tensor )

Load the flattened prompt embeddings saved by PEFT (prompt_embeddings).

PEFT saves prompt-learning adapters as a single prompt_embeddings tensor. For CARTRIDGE, we split that tensor into frozen and trainable segments according to self.num_frozen_tokens.

Update on GitHub