40 Tokens per Second, Zero Words

I recently ran into a failure mode that looked impossible at first: a multimodal model was generating at roughly 40 tokens per second, GPU KV-cache usage was growing linearly, the API returned HTTP 200, and yet the assistant said absolutely nothing.

The model was a quantized Gemma4 Unified checkpoint served by vLLM. Text-only requests worked. Image requests did not. Streaming returned an empty delta followed by finish_reason="length"; non-streaming returned content=null. If I removed the completion limit, the request could keep running indefinitely.

The GPU was busy. The model was decoding. There were simply no words.

This post is the story of how the bug moved from “maybe the chat template is wrong” to one deterministic Inf in output channel 1215 of the vision projection.

The corresponding vLLM issue is #48231.

The symptom

The first response looked like this:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": null,
      "reasoning": null
    },
    "finish_reason": "length"
  }],
  "usage": {
    "prompt_tokens": 278,
    "completion_tokens": 1024,
    "total_tokens": 1302
  }
}

Meanwhile, vLLM reported perfectly ordinary decode activity:

Avg generation throughput: 38.8 tokens/s
Running: 1 reqs
GPU KV cache usage: 3.0%

That combination matters. A stalled engine would have pointed toward scheduling, CUDA graphs, or an attention kernel. This engine was not stalled. It was producing one token after another.

So the first useful question was not “why is there no text?” but “what tokens are actually being generated?”

Ask for token IDs before guessing

vLLM can return generated token IDs through its OpenAI-compatible API:

response = client.chat.completions.create(
    model="gemma-4-12B-it-qat-w4a16-ct",
    messages=messages,
    max_completion_tokens=1024,
    extra_body={"return_token_ids": True},
)

print(response.choices[0].token_ids[:32])

The result was unambiguous:

[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...]

All 1024 completion tokens were ID 0.

This immediately ruled out one class of bugs. The API server was not dropping valid text during streaming or response assembly. The engine itself was repeatedly selecting the same special token.

I still did not know whether token 0 was the cause or merely the symptom.

The plausible explanations that were wrong

Gemma4 has several moving parts around generation: a custom chat template, optional thinking, multimodal sentinel tokens, reasoning parsing, and tool-call parsing. Any of them could plausibly produce an empty visible response.

I tested the obvious possibilities:

  • thinking enabled and disabled;
  • streaming and non-streaming requests;
  • bad_words=["<pad>"];
  • logit_bias={"0": -100};
  • FlashInfer sampling disabled;
  • speculative decoding disabled;
  • a 1x1 PNG, a generated 512x512 PNG, and a 1920x1080 image.

Nothing changed.

The 1x1 image did trigger a Transformers warning about an ambiguous channel dimension, which was initially suspicious. But normal-resolution images failed in exactly the same way. The warning was noise.

The prompt itself was also correct. Returned prompt IDs contained the image boundary tokens and 280 expanded image placeholders:

... 255999, 258880, 258880, ... 258880, 258882 ...

At this point, token suppression was becoming the wrong abstraction. If the model assigned pathological scores to the vocabulary, banning token 0 would only force it to choose a different pathological token.

It was time to inspect the logits.

The sampler received no finite logits

The first instrumentation went immediately after vLLM converted logits to FP32 in the sampler:

raw_nan_counts = torch.isnan(logits).sum(dim=-1)
raw_finite_counts = torch.isfinite(logits).sum(dim=-1)

For the real request:

shape=(1, 262144)
raw_nan_counts=[262135]
raw_finite_counts=[0]

There were no finite logits at all.

That explained why bad_words and logit_bias were ineffective. Those features operate on meaningful logits. They cannot recover a probability distribution from NaNs.

It also reframed token ID 0: the sampler was not confidently choosing padding. It was collapsing in the presence of an invalid score vector.

The next question was where the NaNs first appeared.

Walking backward from the LM head

I added small diagnostics around compute_logits, taking care not to scan the entire LM-head weight on the GPU. An earlier attempt to call torch.isfinite() on the full matrix allocated several gigabytes and caused an out-of-memory error during startup. An 8x8 sample was enough to distinguish a corrupted weight tensor from a corrupted hidden state.

Warmup looked healthy. The actual request did not:

hidden={'shape': (1, 3840), 'nan': 3840, 'finite': 0}
lm_head_weight={
  'shape': (262144, 3840),
  'sample_nan': 0,
  'sample_finite': 64
}
logits={'shape': (1, 262144), 'nan': 262144, 'finite': 0}

The LM head was not creating the NaNs. Its input was already entirely NaN.

Moving one boundary earlier, I logged the multimodal embeddings before the language model and the hidden states after it:

inputs_embeds={
  'shape': (278, 3840),
  'nan': 983040,
  'finite': 84480
}
hidden={
  'shape': (278, 3840),
  'nan': 1067520,
  'finite': 0
}

The image embeddings entered the language model already contaminated. The decoder merely propagated them.

That left the encoder-free vision embedder.

One stage at a time

Gemma4 Unified does not use a SigLIP vision tower in this path. Raw image patches go through a relatively short pipeline:

patches -> LayerNorm -> Dense -> LayerNorm
        -> positional embedding -> LayerNorm

That is an unusually friendly structure to debug. I recorded finite, NaN, and Inf counts after every operation:

cast_pixels = pixel_values.to(self.pos_embedding.dtype)
after_ln1 = self.patch_ln1(cast_pixels)
after_dense, _ = self.patch_dense(after_ln1)
after_ln2 = self.patch_ln2(after_dense)
pos_embs = self._factorized_posemb(pixel_position_ids)
hidden_states = self.pos_norm(after_ln2 + pos_embs)

The output finally exposed the transition:

pixels: dtype=float16, nan=0, inf=0, bad_rows=0
ln1:    dtype=float16, nan=0, inf=0, bad_rows=0
dense:  dtype=float16, nan=0, inf=256, bad_rows=256
ln2:    dtype=float16, nan=983040, inf=0, bad_rows=256
posemb: dtype=float16, nan=0, inf=0, bad_rows=0

The input pixels were fine. The first LayerNorm was fine. The Dense projection produced exactly 256 infinities, one for every valid image patch. The following LayerNorm converted each affected row into 3840 NaNs:

256 * 3840 = 983040

Even better, all 256 infinities appeared in the same output channel:

dense_inf_locations=[
  [0, 1215], [1, 1215], ... [255, 1215]
]
dense_finite_absmax=49248.0

The bug was deterministic and local: output channel 1215 of vision_embedder.patch_dense overflowed for every valid patch.

Why BF16 worked and FP16 did not

The server had printed an important warning during model loading:

Your device 'NVIDIA GeForce RTX 2080 Ti' (with compute capability 7.5)
doesn't support torch.bfloat16. Falling back to torch.float16 for compatibility.
Casting torch.bfloat16 to torch.float16.

The checkpoint declared BF16, but Turing GPUs do not provide native BF16 support. vLLM therefore selected FP16.

BF16 and FP16 both occupy 16 bits, but they spend those bits differently. BF16 keeps the eight-bit exponent of FP32 and has a maximum magnitude around 3e38. FP16 has more mantissa precision but only a five-bit exponent, with a maximum finite value of 65504.

The vision projection was excluded from W4A16 quantization. Its stored BF16 tensors were finite, but their range was striking:

patch_dense.weight: min=-130, max=147
patch_dense.bias:   min=-127, max=126

The projection has an input dimension of 6912. Once those values were evaluated in FP16, one output channel crossed 65504 and became Inf.

An important subtlety is that FP32 accumulation does not necessarily save this computation. Even if the matrix multiplication accumulates internally in FP32, storing its final output as FP16 still overflows. The LayerNorm that could have reduced the magnitude comes one operation too late.

And AMP does not automatically rescale inference activations. GradScaler solves a different problem during training.

A useful proof, but not the final fix

The first successful workaround computed a scaled affine result:

(Wx + b) / 256

Because LayerNorm immediately follows the projection, multiplying the complete affine result by a positive constant is almost invariant, apart from the epsilon term. The workaround removed the overflow and restored normal output.

It was also useful evidence: once the Dense result stayed finite, every downstream tensor became finite.

But an empirical scale factor is not an ideal production fix. It introduces a magic number, and tensor parallelism requires applying the scaled bias to each local output shard before all-gather.

The cleaner solution is to preserve the required dynamic range directly.

The selective-FP32 fix

The final patch keeps only the overflow-prone vision projection and its adjacent LayerNorm in FP32 when the platform does not support the checkpoint dtype:

checkpoint_dtype = getattr(config, "dtype", None)
use_fp32_projection = (
    checkpoint_dtype is not None
    and checkpoint_dtype not in current_platform.supported_dtypes
)

projection_dtype = (
    torch.float32
    if use_fp32_projection
    else torch.get_default_dtype()
)

The projection and normalization use that dtype:

self.patch_dense = ColumnParallelLinear(
    patch_dim,
    mm_embed_dim,
    bias=True,
    params_dtype=projection_dtype,
    quant_config=quant_config,
    prefix=f"{prefix}.patch_dense",
    gather_output=True,
)
self.patch_ln2 = nn.LayerNorm(mm_embed_dim, dtype=projection_dtype)

In the fallback path, the input is promoted before the affine operation and demoted only after normalization:

hidden_states = self.patch_ln1(pixel_values.to(self.pos_embedding.dtype))
if use_fp32_projection:
    hidden_states = hidden_states.float()

hidden_states, _ = self.patch_dense(hidden_states)
hidden_states = self.patch_ln2(hidden_states)

if use_fp32_projection:
    hidden_states = hidden_states.to(self.pos_embedding.dtype)

This is deliberately local:

  • the language model remains W4A16;
  • the rest of the multimodal path remains in the normal model dtype;
  • platforms that support the checkpoint dtype keep their original behavior;
  • the existing ColumnParallelLinear(gather_output=True) tensor-parallel path remains intact.

The model does not need to run globally in FP32. Only the operation that requires the extra exponent range does.

After the fix

The same checkpoint and image request produced:

vision embedded: 1,075,200 / 1,075,200 finite
valid features:    983,040 /   983,040 finite
LM hidden:       1,067,520 / 1,067,520 finite
decode logits:     262,144 /   262,144 finite
NaN values: 0

The API returned a normal description of the image. The sampler stopped producing token ID 0. Generation terminated normally.

The numerical chain was now the one we wanted:

finite pixels
  -> finite FP32 vision projection
  -> finite FP32 LayerNorm
  -> finite model-dtype image embeddings
  -> finite hidden states
  -> finite logits
  -> normal text

What I would do earlier next time

This investigation took several restarts, including a few self-inflicted failures from diagnostics that were too expensive or were executed during model profiling. The useful lessons are fairly general.

Return token IDs early

An empty API response does not imply empty generation. Token IDs separated the serving layer from the model path immediately.

Follow non-finite values backward

Once the sampler showed no finite logits, token suppression was no longer a meaningful fix. The fastest route was to move backward through clear tensor boundaries: logits, hidden states, multimodal embeddings, then each vision stage.

Distinguish warmup from real requests

The profiling run was finite. Looking only at the first diagnostic line would have falsely cleared the model. Logging every invocation until the real request appeared made the difference.

Avoid full-tensor diagnostics on large weights

Computing torch.isfinite() over a 262144 x 3840 LM head created a large temporary tensor and caused an OOM. Sample weights; inspect full activations only when their size is controlled.

One Inf before LayerNorm is enough

LayerNorm does not quarantine a bad channel. A single infinity can contaminate the full row, which then contaminates attention and every later token.

Treat sampler collapse as a symptom

Masking token ID 0 and applying nan_to_num changed the symptom but could not restore semantic output. Invalid logits must be fixed at their source.

Closing thought

The strangest part of this bug was how healthy the system looked from the outside. Throughput was normal. CUDA kernels were running. KV cache was filling. HTTP requests completed successfully.

The model was doing a great deal of computation to produce no information.

In numerical systems, “running” and “working” are very different states. Sometimes the shortest path between them is one counter placed after every layer.

An unhandled error has occurred. Reload âś•

Rejoining the server...

Rejoin failed... trying again in seconds.

Failed to rejoin.
Please retry or reload the page.

The session has been paused by the server.

Failed to resume the session.
Please retry or reload the page.