40 Tokens per Second, Zero Words: Debugging a Gemma4 Vision Overflow in vLLM

I had a multimodal model generating at roughly 40 tokens per second. Its GPU KV-cache usage grew linearly and the API returned HTTP 200, yet the assistant produced no text.

The model was a quantized Gemma4 Unified checkpoint served by vLLM. Text-only requests worked, but image requests did not. Streaming returned an empty delta followed by finish_reason="length", while non-streaming returned content=null. Without a completion limit, the request could run indefinitely.

The GPU stayed busy because the model was decoding tokens. The failure eventually led to one deterministic Inf in output channel 1215 of the vision projection, not the chat template I first suspected.

The corresponding vLLM issue is #48231.

The symptom

The first response looked like this:

{
  "choices": [{
    "message": {
      "role": "assistant",
      "content": null,
      "reasoning": null
    },
    "finish_reason": "length"
  }],
  "usage": {
    "prompt_tokens": 278,
    "completion_tokens": 1024,
    "total_tokens": 1302
  }
}

Meanwhile, vLLM reported perfectly ordinary decode activity:

Avg generation throughput: 38.8 tokens/s
Running: 1 reqs
GPU KV cache usage: 3.0%

This combination ruled out a stalled engine, which would have pointed toward scheduling, CUDA graphs, or an attention kernel. The engine was producing one token after another.

I needed to find out which tokens it was generating.

Ask for token IDs before guessing

vLLM can return generated token IDs through its OpenAI-compatible API:

response = client.chat.completions.create(
    model="gemma-4-12B-it-qat-w4a16-ct",
    messages=messages,
    max_completion_tokens=1024,
    extra_body={"return_token_ids": True},
)

print(response.choices[0].token_ids[:32])

The response contained the same ID throughout:

[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...]

All 1024 completion tokens were ID 0. The API server was not dropping valid text during streaming or response assembly. The engine itself kept selecting the same special token.

At this stage, token 0 could still have been either the cause or a symptom.

The plausible explanations that were wrong

Gemma4 generation involves a custom chat template, optional thinking, multimodal sentinel tokens, reasoning parsing, and tool-call parsing. Any of these could have hidden an otherwise valid response.

I tested the obvious possibilities:

  • thinking enabled and disabled;
  • streaming and non-streaming requests;
  • bad_words=["<pad>"];
  • logit_bias={"0": -100};
  • FlashInfer sampling disabled;
  • speculative decoding disabled;
  • a 1x1 PNG, a generated 512x512 PNG, and a 1920x1080 image.

Every test produced the same result.

The 1x1 image triggered a Transformers warning about an ambiguous channel dimension. Normal-resolution images failed in exactly the same way, so the warning was unrelated.

The prompt itself was also correct. Returned prompt IDs contained the image boundary tokens and 280 expanded image placeholders:

... 255999, 258880, 258880, ... 258880, 258882 ...

Token suppression could not explain the failure. If the model assigned pathological scores across the vocabulary, banning token 0 would only make it choose another bad token. I inspected the logits instead.

The sampler received no finite logits

The first instrumentation went immediately after vLLM converted logits to FP32 in the sampler:

raw_nan_counts = torch.isnan(logits).sum(dim=-1)
raw_finite_counts = torch.isfinite(logits).sum(dim=-1)

For the real request:

shape=(1, 262144)
raw_nan_counts=[262135]
raw_finite_counts=[0]

There were no finite logits. That explained why bad_words and logit_bias had no effect: both operate on meaningful logits and cannot recover a probability distribution from NaNs.

The sampler was not confidently choosing padding. It was collapsing on an invalid score vector and returning token ID 0.

I then traced where the NaNs first appeared.

Walking backward from the LM head

I added small diagnostics around compute_logits without scanning the entire LM-head weight on the GPU. Calling torch.isfinite() on the full matrix had already allocated several gigabytes and caused an out-of-memory error during startup. An 8x8 sample was enough to distinguish a corrupted weight tensor from a corrupted hidden state.

Warmup looked healthy. The actual request did not:

hidden={'shape': (1, 3840), 'nan': 3840, 'finite': 0}
lm_head_weight={
  'shape': (262144, 3840),
  'sample_nan': 0,
  'sample_finite': 64
}
logits={'shape': (1, 262144), 'nan': 262144, 'finite': 0}

The LM head received an input that was already entirely NaN.

Moving one boundary earlier, I logged the multimodal embeddings before the language model and the hidden states after it:

inputs_embeds={
  'shape': (278, 3840),
  'nan': 983040,
  'finite': 84480
}
hidden={
  'shape': (278, 3840),
  'nan': 1067520,
  'finite': 0
}

The image embeddings entered the language model with NaNs, which the decoder propagated.

The remaining source was the encoder-free vision embedder.

One stage at a time

Gemma4 Unified does not use a SigLIP vision tower in this path. Raw image patches go through a relatively short pipeline:

patches -> LayerNorm -> Dense -> LayerNorm
        -> positional embedding -> LayerNorm

The short pipeline made stage-by-stage checks practical. I recorded finite, NaN, and Inf counts after every operation:

cast_pixels = pixel_values.to(self.pos_embedding.dtype)
after_ln1 = self.patch_ln1(cast_pixels)
after_dense, _ = self.patch_dense(after_ln1)
after_ln2 = self.patch_ln2(after_dense)
pos_embs = self._factorized_posemb(pixel_position_ids)
hidden_states = self.pos_norm(after_ln2 + pos_embs)

The counts located the transition:

pixels: dtype=float16, nan=0, inf=0, bad_rows=0
ln1:    dtype=float16, nan=0, inf=0, bad_rows=0
dense:  dtype=float16, nan=0, inf=256, bad_rows=256
ln2:    dtype=float16, nan=983040, inf=0, bad_rows=256
posemb: dtype=float16, nan=0, inf=0, bad_rows=0

The input pixels and first LayerNorm output were finite. The Dense projection produced exactly 256 infinities, one for every valid image patch. The following LayerNorm converted each affected row into 3840 NaNs:

256 * 3840 = 983040

All 256 infinities appeared in the same output channel:

dense_inf_locations=[
  [0, 1215], [1, 1215], ... [255, 1215]
]
dense_finite_absmax=49248.0

The result was deterministic and local: output channel 1215 of vision_embedder.patch_dense overflowed for every valid patch.

Why BF16 worked and FP16 did not

During model loading, the server had printed this warning:

Your device 'NVIDIA GeForce RTX 2080 Ti' (with compute capability 7.5)
doesn't support torch.bfloat16. Falling back to torch.float16 for compatibility.
Casting torch.bfloat16 to torch.float16.

The checkpoint declared BF16, but Turing GPUs do not provide native BF16 support, so vLLM selected FP16.

BF16 and FP16 both occupy 16 bits, but divide those bits differently. BF16 keeps the eight-bit exponent of FP32 and has a maximum magnitude around 3e38. FP16 has more mantissa precision, a five-bit exponent, and a maximum finite value of 65504.

The vision projection was excluded from W4A16 quantization. Its stored BF16 tensors were finite, with this range:

patch_dense.weight: min=-130, max=147
patch_dense.bias:   min=-127, max=126

The projection has an input dimension of 6912. Evaluating those values in FP16 pushed one output channel past 65504 and produced Inf.

FP32 accumulation does not necessarily prevent this overflow. Even if the matrix multiplication accumulates internally in FP32, its final FP16 output can still exceed the representable range. The LayerNorm that could reduce the magnitude runs one operation too late.

AMP does not automatically rescale inference activations. GradScaler addresses a different problem during training.

A useful proof, but not the final fix

My first successful workaround computed a scaled affine result:

(Wx + b) / 256

LayerNorm immediately follows the projection, so multiplying the complete affine result by a positive constant is almost invariant apart from the epsilon term. This removed the overflow and restored normal output.

It also confirmed the diagnosis. Once the Dense result stayed finite, every downstream tensor did too.

An empirical scale factor is a poor production fix. It introduces a magic number, and tensor parallelism requires the scaled bias on each local output shard before all-gather.

Preserving the required dynamic range avoids both problems.

The selective-FP32 fix

The final patch keeps the overflow-prone vision projection and its adjacent LayerNorm in FP32 when the platform does not support the checkpoint dtype:

checkpoint_dtype = getattr(config, "dtype", None)
use_fp32_projection = (
    checkpoint_dtype is not None
    and checkpoint_dtype not in current_platform.supported_dtypes
)

projection_dtype = (
    torch.float32
    if use_fp32_projection
    else torch.get_default_dtype()
)

The projection and normalization then use that dtype:

self.patch_dense = ColumnParallelLinear(
    patch_dim,
    mm_embed_dim,
    bias=True,
    params_dtype=projection_dtype,
    quant_config=quant_config,
    prefix=f"{prefix}.patch_dense",
    gather_output=True,
)
self.patch_ln2 = nn.LayerNorm(mm_embed_dim, dtype=projection_dtype)

On the fallback path, the input is promoted before the affine operation and demoted after normalization:

hidden_states = self.patch_ln1(pixel_values.to(self.pos_embedding.dtype))
if use_fp32_projection:
    hidden_states = hidden_states.float()

hidden_states, _ = self.patch_dense(hidden_states)
hidden_states = self.patch_ln2(hidden_states)

if use_fp32_projection:
    hidden_states = hidden_states.to(self.pos_embedding.dtype)

The change remains local:

  • the language model remains W4A16;
  • the rest of the multimodal path remains in the normal model dtype;
  • platforms that support the checkpoint dtype keep their original behavior;
  • the existing ColumnParallelLinear(gather_output=True) tensor-parallel path remains intact.

The model does not need to run globally in FP32. Only this operation needs the extra exponent range.

After the fix

The same checkpoint and image request produced:

vision embedded: 1,075,200 / 1,075,200 finite
valid features:    983,040 /   983,040 finite
LM hidden:       1,067,520 / 1,067,520 finite
decode logits:     262,144 /   262,144 finite
NaN values: 0

The API returned a normal image description, the sampler stopped producing token ID 0, and generation terminated normally.

The numerical chain was now finite throughout:

finite pixels
  -> finite FP32 vision projection
  -> finite FP32 LayerNorm
  -> finite model-dtype image embeddings
  -> finite hidden states
  -> finite logits
  -> normal text

What I would check first next time

This investigation required several restarts, some caused by diagnostics that were too expensive or ran during model profiling. Six checks would shorten a repeat investigation.

Return token IDs early. An empty API response does not imply empty generation. Token IDs separated the serving layer from the model path immediately.

Follow non-finite values backward. Once the sampler showed no finite logits, token suppression was no longer meaningful. The direct route was through clear tensor boundaries: logits, hidden states, multimodal embeddings, and then each vision stage.

Distinguish warmup from real requests. The profiling run was finite, so the first diagnostic line could have falsely cleared the model. I needed logs from every invocation through the real request.

Avoid full-tensor diagnostics on large weights. Computing torch.isfinite() over a 262144 x 3840 LM head created a large temporary tensor and caused an OOM. Sampling weights is enough; full activation scans are appropriate only when their size is controlled.

One Inf before LayerNorm is enough. LayerNorm does not quarantine a bad channel. A single infinity can contaminate the full row, attention, and every later token.

Treat sampler collapse as a symptom. Masking token ID 0 and applying nan_to_num changed the visible failure but could not restore semantic output. The fix belonged at the source of the invalid logits.

The external health signals were misleading throughout. Throughput was normal, CUDA kernels were running, KV cache was filling, and HTTP requests completed successfully. Per-layer finite-value counts exposed what those metrics could not: the model was running, but one overflowing channel prevented it from producing information.

An unhandled error has occurred. Reload ✕

Rejoining the server...

Rejoin failed... trying again in seconds.

Failed to rejoin.
Please retry or reload the page.

The session has been paused by the server.

Failed to resume the session.
Please retry or reload the page.