TLDR: tool parameters using the common JSON Schema pattern `anyOf: [$ref, null]` are rendered into the prompt as empty `type` fields. This strips the useful schema information before the model sees it.
--
Long, rambling version:
Gemma 4 was having issues with calling my custom MCP tool on >3 inference engines, while Qwen3.5 and gpt-oss-20b were doing fine.
I guessed it was either a chat template issue or inference library issue on an edge case, and thought time would sort it out, since many people were happy with Gemma 4 as an agent.
It didn't for at least 2 weeks now and I had no choice but to investigate myself.
What I did:
- I made a verbose log file via llama-server, running the same prompt/tool on Qwen3.5-27B-Q4_K_M and gemma-4-31B-it-Q4_K_S on a macbook pro.
- I asked GPT-5.5-high on codex CLI to read the logs and diagnose the issue.
- Found it in couple of minutes; the default Gemma chat template assumes tool parameters have a direct type field. Which means it will not work with JSON schema shapes like nullable refs:
{"anyOf": [{"$ref": "#/$defs/SomeObject"}, {"type": "null"}]}
where there is no top-level type. The useful structure is inside anyOf and $defs. The template drops anyOf, $ref, and $defs, then renders it as type: "".
- It was fixed by small changes in the chat template jinja, and now Gemma is calling my tool perfectly!
Anyway I made a PR on HF, google/gemma-4-31B-it.
<UPDATE>
I realized that I just addressed one of many issues arising JSON Schema shapes that do not expose their meaning through a direct top-level type. I updated the jinja to address:
now the jinja preserves:
- $ref
- anyOf
- oneOf
- allOf
- $defs
- enum
- const
- type: ["string", "null"]
- items / properties for type arrays containing array or object
- null values as null
The fixed jinja:
https://pastebin.com/tBAHN6FV
[link] [comments]
