CLIP's contrastive pretraining matched a fully-supervised ResNet-50 on zero-shot ImageNet, 76.2% against the prior best zero-shot result of 11.5%, using no labels at all, and every vision-language model since has picked one of exactly two real ways to plug that visual signal into a language model: LLaVA's single projection layer or Flamingo's gated cross-attention, and Qwen2-VL and Llama 3.2 Vision are real, live, frontier-lab proof that both choices still ship today

Every post in this series so far has been about a model that only ever sees text. The transformer itself, its attention mechanism, the data pipelines that feed it, all of it assumes tokens are the only kind of input there is. Real frontier models don’t get to assume that: GPT-4V, Gemini, Qwen2-VL, and Llama 3.2 Vision all take an image (and increasingly audio and video) as first-class input, and none of them do it by inventing a new kind of transformer. They all solve the same underlying problem, get a non-text signal into a space a language model’s attention can operate over, and the real, disclosed history of this field is that there are exactly two structurally different answers to that problem, both still in active production use, and the choice between them is a genuine architectural decision with real, different consequences, not a detail.

CLIP: the contrastive pretraining underneath almost everything here#

Before any model can generate text about an image, something has to learn that an image and a sentence describing it belong together. Radford et al.’s 2021 CLIP paper solved this with a deliberately simple idea: train an image encoder and a text encoder jointly, with no labels at all, purely by contrastive learning against real (image, caption) pairs scraped from the web. For a batch of $N$ (image, text) pairs, encode every image and every text, then push each image’s embedding toward its own caption’s embedding and away from the other $N-1$ captions in the same batch, symmetrically in both directions:

$\mathcal{L} = -\frac{1}{2N}\sum_{i=1}^{N}\left[\log\frac{\exp(\text{sim}(I_i,T_i)/\tau)}{\sum_{j=1}^{N}\exp(\text{sim}(I_i,T_j)/\tau)} + \log\frac{\exp(\text{sim}(I_i,T_i)/\tau)}{\sum_{j=1}^{N}\exp(\text{sim}(T_i,I_j)/\tau)}\right]$

where $\text{sim}$ is cosine similarity and $\tau$ is a learned temperature, not a fixed hyperparameter, the model itself decides how sharply to separate the true pair from the batch’s negatives. The real, load-bearing detail is the batch size: CLIP trained with a global batch of 32,768, meaning every single positive pair is contrasted against 32,767 real negatives drawn from the same batch, at once. This is exactly why CLIP-style pretraining is expensive in a way that has nothing to do with model depth, the loss’s quality scales with how many negatives you can hold in one batch, which is a memory and communication problem, not a parameter-count one.

The real result this bought: 76.2% zero-shot top-1 accuracy on ImageNet, using no ImageNet labels whatsoever, against the best prior zero-shot result of 11.5% from earlier visual n-gram work, and landing in the same range as a fully-supervised ResNet-50. That gap, 11.5% to 76.2% from a change in training objective alone, is the real reason CLIP’s image and text encoders became the default starting point for almost every vision-language model built after it: the encoders arrive already knowing which images and which words belong together, before a single token of instruction-tuning data exists.

Two real designs for getting that signal into an LLM#

CLIP gives you an image encoder. It does not, by itself, give you a model that can answer a question about an image in fluent text. That requires connecting a vision encoder’s output to a language model’s input, and the field’s real, disclosed history settled on two structurally different answers.

LLaVA: one projection layer, and almost nothing else#

Liu et al.’s LLaVA takes the simpler path: feed CLIP’s visual features through a single trainable linear layer (upgraded to a small MLP in the Improved Baselines follow-up) that projects them into the LLM’s own token-embedding space, then literally prepend the resulting vectors to the text token sequence as if they were more tokens. Everything downstream, every layer of self-attention, treats image and text tokens identically; there’s no architectural distinction between them once the projection has run. The real, disclosed training numbers are worth having exactly because they’re surprising: the original LLaVA trained on 600K image-text pairs, the improved baselines version on 1.2M, and the full training run, projector plus a light instruction-tuning pass, finishes in about one day on a single 8-A100 node. The paper’s own stated finding is blunt: this “surprisingly powerful and data-efficient” connector, one linear layer, is enough, and the real capability gain from LLaVA to its successors has come mostly from better data curation, not a more elaborate fusion mechanism.

Flamingo: keep both backbones frozen, insert new cross-attention#

Alayrac et al.’s Flamingo (DeepMind, 2022) takes the opposite bet: keep the pretrained vision encoder and the pretrained language model both completely frozen, and insert new, trainable layers between the LLM’s existing blocks rather than routing visual information through the input sequence at all. Two new components do the work. A Perceiver Resampler compresses a variable number of visual features (which changes with image resolution and, for video, frame count) down to a small, fixed number of latent vectors, so the rest of the model never has to handle a variable-length visual input. Those fixed latents then feed into newly-inserted gated cross-attention dense (GATED XATTN-DENSE) layers, interleaved between the LLM’s own frozen self-attention blocks, where language tokens attend to the visual latents.

The detail worth pausing on, because it’s the same underlying trick this series already met from a different angle: each gated cross-attention layer’s contribution is scaled by a learned gate initialized at zero. At the start of training, that gate makes the new layer’s output exactly zero, meaning the model behaves identically to the frozen, pretrained LLM, with no visual information mixed in at all, and training gradually opens the gate as the new layers learn to contribute something useful. That’s the same initialization principle behind LoRA’s zero-initialized $B$ matrix: whenever you’re inserting a new, trainable component next to a large frozen or pretrained one, starting that component’s contribution at exactly zero is what keeps early training from destabilizing behavior you already paid to earn.

Two frontier labs, two real, live, different answers in 2024#

The reason this is a genuinely open design decision, not a solved problem with one correct answer, is that current frontier models don’t agree on it.

Qwen2-VL and Qwen2.5-VL lean toward the LLaVA family’s philosophy, digest everything into a token sequence and let ordinary self-attention do the fusion, but push it further: a roughly 675M-parameter Vision Transformer with naive dynamic resolution, meaning images of arbitrary resolution and aspect ratio are converted into a variable-length sequence of patches without padding or forced resizing, using 2D-RoPE instead of fixed absolute position embeddings to preserve the image’s actual 2D spatial layout. Qwen2.5-VL adds windowed attention inside the vision encoder itself to keep that flexibility computationally affordable at higher resolutions.

Llama 3.2 Vision leans the other way, toward Flamingo’s philosophy: Meta’s own model card states the text model is initialized from the existing, pretrained Llama 3.1 checkpoint and kept completely frozen throughout vision training. A separately trained adapter inserts new cross-attention layers after every fourth self-attention layer in the language model, specifically at layers 3, 8, 13, 18, 23, 28, 33, and 38, where the language model’s hidden states attend outward to the image encoder’s representations. Only the image encoder and these new adapter layers are updated; the language model’s own weights never move. The stated reason is direct and matches this series’ running theme about catastrophic forgetting: freezing the text model is what preserves Llama 3.1’s original text-only performance exactly, giving developers what Meta calls a drop-in replacement rather than a model that’s quietly worse at plain text than the version without vision.

Neither choice is strictly better; they’re different bets on the same tradeoff. LLaVA-style fusion lets vision and text interact through full self-attention at every layer, which can capture richer cross-modal interaction, at the cost of retraining (or at least fine-tuning) the whole stack and risking exactly the forgetting problem already derived in the previous post. Flamingo-style cross-attention guarantees the frozen backbone’s original behavior is structurally protected, at the cost of a more constrained, less flexible fusion mechanism and real added inference-time cost from the extra cross-attention layers on every forward pass.

The loss function doesn’t change, only what’s in context does#

A detail easy to miss and worth stating plainly: once fusion happens, whichever architecture you pick, the actual training loss for instruction-tuning a vision-language model is identical to the plain-text cross-entropy loss this series already derived. Image tokens (or the visual latents Flamingo produces) sit in the context and are attended over, but the loss is only ever computed on the text tokens the model is meant to generate; the image never appears as a prediction target. The multimodal-ness of a “multimodal model” is entirely about what’s fed into the context and how it’s fused, not about a new objective function. What does change is the data: instruction-tuning now needs interleaved image-text conversational data (a real, disclosed example of the kind of large-scale curated dataset this requires is OBELICS, open web-scale interleaved image-text documents), not just plain text instruction pairs.

Real failure modes, not toy caveats#

• Object hallucination. A vision-language model describing objects, attributes, or relationships that are not actually present in the image is a well-documented, benchmarked failure mode (POPE is the standard polling-based probe for it), and it’s a direct consequence of the language model’s own strong text-only priors overriding weak or ambiguous visual evidence, exactly the kind of failure you’d expect from a frozen, powerful LLM attending to a comparatively small amount of new visual signal.
• Forgetting vision ability during later text-only fine-tuning. The catastrophic forgetting result from the previous post, that further fine-tuning degrades capabilities the base training established, applies here in a modality-specific way: fine-tuning a vision-language model on text-only data without remixing in any image-text data measurably degrades its vision performance, the same mechanism, a different axis.
• Resolution eats your context budget. This series already derived the real KV-cache cost of a long context window, and image tokens are not free: a higher-resolution image becomes more vision tokens (quadratically more, for a naive fixed-patch encoder), which is real context budget an LLaVA-style model can no longer spend on text, the exact reason Qwen2-VL’s dynamic-resolution and windowed-attention design exists as a real, motivated engineering response rather than an academic flourish.
• Video is the same problem, compounded. Extending any of these designs to video means either sampling frames (and multiplying the token cost by frame count) or building a temporal-aware resampler; the Perceiver Resampler’s fixed-output-size design is precisely what lets Flamingo-style architectures absorb a variable number of video frames without the token count exploding the way a naive per-frame LLaVA-style concatenation would.

Try it yourself#

Beginner. Using CLIP’s contrastive loss above, explain in your own words why a larger batch size makes the pretraining signal stronger, in terms of what each image is being contrasted against. What happens to the loss’s difficulty as batch size shrinks toward 1?

Intermediate. Flamingo’s cross-attention gate is initialized at zero, identical in spirit to LoRA’s zero-initialized $B$ matrix. Write out, symbolically, what the output of a single GATED XATTN-DENSE layer is at initialization, and confirm algebraically that the rest of the frozen LLM behaves exactly as if that layer weren’t there at all.

Advanced. Qwen2-VL’s naive dynamic resolution converts an image into a variable-length patch sequence rather than resizing to a fixed size. Sketch, in pseudocode, how you’d compute the number of vision tokens produced for a given image resolution and patch size, and then compute how many vision tokens a 1024x1024 image produces versus a 336x336 image at a 14x14 patch size, the same comparison that motivates windowed attention inside the vision encoder.

What this takes to be frontier-job-ready#

Real, live job postings treat multimodal work as a core, expected competency rather than a specialization bolted onto text-only roles. Anthropic’s own posting lists, verbatim, among the day-to-day responsibilities: “making a multimodal dataset in a format models can easily consume,” listed alongside pretraining data mixing and scaling-law work, not as a separate track. OpenAI’s pretraining posting states its technical bar as experience “developing or scaling pretraining architectures (LLMs, diffusion models, multimodal models, etc),” treating multimodal architecture work as one item in the same list as core LLM pretraining. And Mistral’s Research Engineer posting describes its Embedded RE team structure as engineers who “sit inside a research squad (Alignment, Pre-training, Multimodal, …),” naming Multimodal as a standing research squad on equal footing with alignment and pretraining itself. None of these treat “add vision” as an afterthought.

---

The one-sentence version: CLIP’s contrastive pretraining, a learned-temperature loss run over a real 32,768-example batch, gets an image encoder and a text encoder to agree on what belongs together well enough to reach 76.2% zero-shot ImageNet accuracy with zero labels; getting that signal into a language model has exactly two real, still-competing answers, LLaVA’s single projection layer treating vision as just more tokens for self-attention to fuse, or Flamingo’s frozen backbone plus zero-initialized gated cross-attention that structurally protects the LLM’s original behavior, and Qwen2-VL and Llama 3.2 Vision are live, 2024, frontier-lab proof that reasonable teams still land on opposite sides of that same choice.