Architectural Sampling: Test-Time Scaling via Computational Diversity in Frozen VLMs

Abstract

Test-time scaling often seeks better answers by sampling multiple responses from a frozen model, yet conventional temperature sampling generates every candidate along the same fixed computation path.

We introduce architectural sampling, a training-free method that generates candidates through distinct forward computations by reusing selected blocks of decoder layers. Varying the block location and repetition count introduces computational diversity without updating model weights or adding auxiliary parameters. Across five Qwen checkpoints and twelve multimodal benchmarks, architectural sampling improves pass@9 over standard-path temperature sampling by 6.58 percentage points on average at the same nine-candidate budget. Reusing early layers yields the strongest gains, and the improvement in candidate coverage persists even under greedy decoding. The resulting candidates show lower lexical overlap and improve accuracy when used as rollouts for label-free test-time reinforcement learning.

Two 3D answer-space landscapes. Left: temperature sampling sends three trajectories from the same starting point to the same dog-region peak, producing near-identical answers. Right: architectural sampling sends four trajectories along different computation paths to different regions, producing answers like 'a golden retriever' and 'a dog in the mountain'. — **Two sources of candidate diversity.** Temperature sampling (left) draws several outputs from the same computation path. Architectural sampling (right) changes the computation path itself, starting from the same multimodal representation $h^{(0)}$, and produces one candidate per path. The landscape is a conceptual illustration of the answer space.

How it works

Temperature only acts at the very last step: every candidate passes through the same layers with the same intermediate representations, and only the final draw differs. Architectural sampling moves the source of diversity upstream, into the computation.

For each candidate we pick a short contiguous window of decoder layers $B_{[a:b]} = f_b \circ \cdots \circ f_a$, apply it $r$ times, then resume the original forward pass. The prefix and suffix stay untouched:

$$h^{(L)} = \underbrace{f_L \circ \cdots \circ f_{b+1}}_{\text{suffix (fixed)}} \circ \underbrace{\color{#8E7CC3}{\big(B_{[a:b]}\big)}^{\color{#3F72C4}{\circ r}}}_{\text{window} \times r} \circ \underbrace{f_{a-1} \circ \cdots \circ f_1}_{\text{prefix (fixed)}}\big(h^{(0)}\big)$$

Setting $r = 1$ recovers the standard forward pass exactly. The weights stay frozen, nothing is trained, and each path can still be decoded greedily or with temperature, so architectural and token-level variation combine.

Diagram: an image and question are encoded into h0, which is fed into a standard VLM with layers f1 to fL and three effective VLMs, each repeating a different highlighted window of layers r times, producing different candidate answers such as 'A dog', 'A brown and white dog', 'A Labrador', and 'A canine'. — **Architectural sampling from a frozen VLM.** The standard path applies every decoder layer once. Each architectural path repeats one contiguous window $r$ times before continuing through the remaining layers. All paths share the same frozen weights.

Results

At an equal budget of nine candidates and the same decoding temperature ($T = 0.6$), architectural sampling raises pass@9 by 6.58 points on average and wins in 55 of 60 checkpoint–benchmark pairs. Choose a model to compare the two candidate pools.

Temperature samplingArchitectural samplingpass@9, %

Six line charts of pass@k from k=1 to 9 for Qwen3-VL-8B on AI2D, A-OKVQA, BLINK, CV-Bench, MMMU and ScienceQA. The architectural sampling curve keeps rising while temperature sampling flattens early. — **Coverage keeps growing with the candidate budget.** pass@k of Qwen3-VL-8B as k grows. Temperature sampling plateaus early; architectural sampling keeps finding new correct answers.

What makes it work

Early layers matter most

Every window placement beats temperature sampling, but gains shrink as the window moves deeper. Early windows score highest on all five ablation benchmarks.

Reprocess both modalities

Updating visual and language tokens together works best. Language-only updates keep most of the gain; visual-only updates keep less.

It works even with greedy decoding

At $T = 0$, repeated calls to the standard model return the same answer. Architectural candidates stay diverse, and pass@9 is stable across temperatures.

Answers overlap less

Mean lexical diversity (1 − Self-BLEU) is 0.51 for architectural candidates versus 0.22 for temperature samples. It also beats beam search, top-k and top-p at the same budget.

Left: pass@9 for five benchmarks stays nearly flat as sampling temperature goes from 0 to 1. Right: distributions of lexical diversity, with temperature samples centered near 0.22 and architectural samples near 0.51. — **Temperature robustness and candidate diversity** on Qwen3-VL-8B. Left: pass@9 holds steady from greedy decoding to $T = 1$. Right: architectural candidates sit at markedly higher lexical diversity.

Better rollouts for test-time learning

We swap the temperature-sampled rollouts in TTRV, a label-free GRPO-based test-time RL method, for architectural candidates. No ground-truth labels are used; rewards come from majority vote over the model's own outputs. Accuracy improves on every benchmark for both backbones.

pass@1 after TTRV	CV-Bench	MMStar	AI2D	A-OKVQA	ScienceQA	CountBench
Qwen3.5-VL-2B
Temperature rollouts	74.60	45.52	75.38	68.04	67.01	93.06
Architectural rollouts	80.29+5.69	49.17+3.66	76.94+1.56	83.38+15.34	77.33+10.32	95.10+2.04
Qwen3-VL-8B
Temperature rollouts	86.17	53.93	76.61	87.40	84.75	93.17
Architectural rollouts	86.29+0.11	55.52+1.59	77.80+1.17	87.95+0.55	88.82+4.07	96.93+3.76

More scaling curves

The same trend holds on other checkpoints, though not uniformly across every panel.

pass@k curves for Qwen2.5-VL-7B on six benchmarks; architectural sampling ends above temperature sampling on all six. — **Qwen2.5-VL-7B**

pass@k curves for Qwen3.5-VL-4B on six benchmarks; temperature sampling leads at small k on some benchmarks, architectural sampling pulls ahead by k=9. — **Qwen3.5-VL-4B**

Limitations

Reusing layers adds forward cost: the nine architectural candidates take about 10.5 standard decoder traversals versus 9.0 for temperature sampling (53.8 s vs. 46.1 s on Qwen3-VL-8B). The method also needs access to internal decoder layers, so it cannot be used with closed models served only through an API.

Citation

@inproceedings{singh2027architectural,
  title     = {Architectural Sampling: Test-Time Scaling via Computational
               Diversity in Frozen Vision-Language Models},
  author    = {Singh, Akshit and Marjit, Shyam and Lin, Wei and
               Karlinsky, Leonid and Mirza, M. Jehanzeb},
  booktitle = {Under review},
  year      = {2027}
}

One checkpoint, nine paths