I gave my LLM 100,000+ tools. Here is what happened

TL;DR: You don't need a massive context window or a giant model to handle an absurd number of tools. By using a Lazy Discovery pattern, a local 4B model (Gemma 4 E4B) successfully solved a massive multi-sector city crisis requiring complex tool navigation, matching Claude Sonnet 4.6 with almost identical efficiency.

The Setup: The "Mega-City Crisis" Benchmark

I wanted to stress-test tool use at an absolute extreme. I simulated a massive infrastructure crisis in a fictional city called Veridian Prime.

• The Scale: ~117,000 registered landmarks/tools split across hierarchical paths (Power, Water, Traffic, Security, etc.).
• The Goal: Find and resolve 4 critical failures while ignoring noise alerts.
• The Catch: One of the failures had a hidden mechanical dependency trap (MECHANICAL_LOCK), meaning the agent had to read an error message, pivot to a completely different infrastructure category to release an emergency brake, and then loop back to finish the job.

I ran this benchmark against two completely different beasts using Elemm (which implements a lazy-loading protocol for tools so the model only pulls what it needs):

1. Gemma 4 E4B (Run locally)
2. Claude Sonnet 4.6 (Run remotely)

Run 1: Gemma 4 E4B (Local)

Verdict: ✅ PASS (17 tool calls)

I honestly expected a local 4B model to choke, but it handled the hierarchy beautifully.

The Good:

• Insane Parallel Batching: It aggressively grouped its inspection commands. It checked all 4 distressed districts at the exact same time.
• Clutched the Trap: When it hit the MECHANICAL_LOCK on the security terminal, it didn’t panic. It read the error, found the release_emergency_brake tool in a different sub-category, executed it, and retried the lockdown—all with zero human intervention.
• Zero Noise Bleed: It completely ignored the low/medium priority noise alerts.

The Jank:

• Minor Action Hallucination: Right after inspecting the districts, it took a "leap of faith" and tried to call non-existent global commands like city:fix_power_surge. Thanks to an on_error: continue fallback policy, it recovered instantly, realized it had to browse the local directory, and found the correct tools.

Run 2: Claude Sonnet 4.6 (Remote)

Verdict: ✅ PASS (19 tool calls)

Sonnet acted exactly like you’d expect a high-tier model to act: highly methodical, extremely cautious, and zero hallucinations.

The Good:

• Clean Syntax: Used native array batching inspect_landmark(["id1", "id2"]) to scan the topology effortlessly.
• Zero Hallucinations: Every single tool call it made was explicitly derived from its structural discovery.
• Resilient: When the server threw a cached state bug on the security logs, Sonnet just shrugged it off and used the status summary to complete the mission.

The Inefficiencies:

• Over-Cautious Diagnostics: Sonnet spent 5 extra tool calls checking system metrics (energy:status, water:pressure) before pulling the trigger. The alert log already told it what was wrong, but Sonnet wanted to double-check. Safe, but slightly higher overhead.

Head-to-Head Comparison

How it works under the hood: The Middleware

If we stuffed 117,000 tool definitions directly into the LLM's system prompt, the context window would have imploded, and the bill would be astronomical.

To solve this, I’m building a custom middleware that exposes a "Lazy Discovery" pattern to the agent.

To put it simply: The middleware exposes a file-system-like directory structure to the LLM using "landmarks". Instead of drowning the model in thousands of tool definitions, the LLM only ever sees a tiny selection of just 8 core tools. These tools handle:

• Navigation: Browsing through the landmark hierarchy.
• Execution Piping: Passing data seamlessly between tool steps.
• Smart Errors + Interactive Help: Providing high-context feedback when something goes wrong (which is exactly how Gemma recovered from its hallucination and how both models figured out the mechanical lock trap).

Because of this architecture, the effective context window at any given second never exceeded a few dozen lines of text.

I will repeat this test after stabilizing the environment, but I trust this process and believe this approach could change how we handle tools for agents. Currently, I am focusing on the ability to load "landmarks" on the fly. With FastAPI, GraphQL, and native Landmarks already on board, this tool can handle a massive number of tools simultaneously, simply by connecting to a URL that presents these files. I will release a new version in the coming days/weeks so you can run this test with your own models. Leave a star on GitHub to stay on track!

Key Takeaway

Seeing a local 4B model solve a multi-step dependency chain across a 100k+ tool library with practically the same efficiency as Sonnet 4.6 proves that smart agent architecture, tailored middleware, and tool-loading protocols matter way more than raw model size for complex automation tasks.

Would love to hear your thoughts! How are you guys handling massive, hierarchical tool environments in your setups?