Your agent ran overnight. One workflow failed halfway through. Three tool calls completed successfully. Two didn't. You're not sure in which order.
What do you actually have to debug with?
For most MCP setups, the honest answer is: not much. Server logs are sparse. Client-side tracing is application-specific. Audit trails are nonexistent. And because MCP interactions happen through a protocol layer, standard API debugging tools don't apply cleanly.
This is the observability gap in production MCP deployments — and it compounds as you scale to multi-agent, multi-server architectures.
Why MCP Observability Is Different
Standard API observability is a solved problem. You instrument the HTTP layer, capture request/response pairs, export to your logging stack, and query when things go wrong.
MCP shifts the model in ways that break this:
Protocol wrapping. Tool calls happen over JSON-RPC or HTTP, but the semantics are richer than a single API endpoint. A tool invocation can chain multiple operations inside the server. The observable boundary shifts inward.
Credential opacity. The calling agent might not know which upstream credentials the server used. If multiple credential modes are active (auto / bring-your-own / server-managed), the audit trail needs to capture which mode fired and with what identity.
Compound action surfaces. Unlike a stateless API endpoint, MCP tools can trigger side effects that accumulate. An agent that loops across a create_issue tool creates multiple issues. Observability isn't just "did the call succeed" — it's "how many downstream effects occurred and are they recoverable."
Session state. MCP servers maintain state across a session. That means observability needs to capture state transitions, not just discrete calls.
The Four Audit Questions
For production MCP, your observability stack needs to answer four questions after any incident:
1. Who called what tool?
- Which agent identity (or user, in multi-tenant setups)
- Which tool name and version
- At what timestamp and with what input parameters
2. What credentials were used?
- Which authentication mode was active
- Which upstream provider was called
- Whether credentials were scoped appropriately for the operation
3. What happened?
- The output or error returned
- Latency and retry behavior
- Whether the operation was idempotent (safe to replay)
4. What side effects occurred?
- Downstream API calls the server made
- Resources created, modified, or deleted
- Spend incurred if execution is metered
Without answers to these four questions, incident response is guesswork.
Logging Patterns That Actually Work
Structured tool call logs
The minimum viable log entry for a tool call:
{
"event": "tool_call",
"tool": "create_file",
"server": "filesystem-server-v1.2",
"session_id": "ses_abc123",
"agent_id": "agent_xyz789",
"timestamp": "2026-04-03T14:32:01Z",
"input_summary": { "path": "/workspace/output.txt", "content_length": 4096 },
"outcome": "success",
"duration_ms": 142,
"idempotent": false,
"side_effects": ["file_created"]
}
The idempotent flag matters. When a retry occurs after a timeout, knowing whether the tool is safe to replay changes your recovery logic entirely.
Error classification
Raw error strings are useless for automated recovery. Structure your error logs:
{
"event": "tool_error",
"tool": "send_email",
"error_class": "auth_expired",
"error_code": "TOKEN_REVOKED",
"recoverable": true,
"recovery_action": "reauth",
"retry_safe": false
}
recoverable tells the orchestrator whether to attempt recovery. retry_safe tells it whether raw retry is safe or risks duplicating the side effect.
Session-level audit trails
Beyond per-call logs, maintain a session summary:
{
"session_id": "ses_abc123",
"started_at": "2026-04-03T14:30:00Z",
"tool_calls": 12,
"successful_calls": 10,
"failed_calls": 2,
"credentials_used": ["fs_local", "openai_byok"],
"side_effects_summary": {
"files_created": 3,
"api_calls_made": 8,
"spend_incurred_usd": 0.042
},
"terminal_state": "partial_success",
"recovery_status": "pending"
}
This session summary is what you need for post-incident analysis, not raw call-level detail.
Cost Attribution in Multi-Tool Agent Loops
When an agent workflow involves multiple MCP servers, spend attribution becomes a real operational concern:
- Which tool consumed which API credits
- Which agent, session, or user incurred which costs
- Whether per-tool spend is within expected bounds
A token-burn governor at the session level prevents runaway spend:
class SpendGovernor:
def __init__(self, session_id: str, limit_usd: float):
self.session_id = session_id
self.limit = limit_usd
self.spent = 0.0
def check(self, estimated_cost: float) -> bool:
if self.spent + estimated_cost > self.limit:
raise SpendLimitExceeded(
f"Session {self.session_id}: limit ${self.limit:.2f} would be exceeded"
)
return True
def record(self, actual_cost: float):
self.spent += actual_cost
Without governors, an agent loop that hits a retry storm on a billable tool can burn real money before the orchestrator notices.
Debugging Partial Failure in MCP Chains
The hardest MCP debugging scenario: a chain of tool calls where some succeeded and some failed, in the middle of the chain.
Your recovery strategy depends on two questions:
Can you find the exact state checkpoint before the failure? If yes, you can resume from the last successful call. If no, you may need to restart the entire workflow.
Are the pre-failure calls reversible? If yes, full rollback is possible. If no — side effects are permanent — your path is forward-only.
Build your workflows to answer both questions explicitly:
- Log a state checkpoint after each successful tool call
- Tag each tool call with its reversibility class:
no_effect | reversible | permanent - On failure, query the most recent state checkpoint before resuming
- Never assume a completed call in one session is visible in a retry session (especially with stateful servers)
What AN Score Captures on Observability
Rhumb's auditability dimension in the production readiness checklist measures this directly. The key signals:
- Structured errors: Does the server return machine-parseable errors with recovery hints, or raw strings?
- Idempotency guarantees: Are tool calls safe to retry without side effect duplication?
- State verification: Is there a mechanism to confirm whether a side effect actually occurred?
- Credential attribution: Does the server expose which auth mode was used on a given call?
High-scoring servers (8.0+) tend to cover all four. Servers below 5.0 often have none. The gap matters most at 2am, when your agent loop has failed partway through and the only thing between you and manual cleanup is your audit trail.
The Observability Checklist
Before promoting an MCP server to production:
- [ ] Tool call logs capture tool name, input summary, outcome, and duration
- [ ] Error logs include error class, recovery hint, and retry-safety flag
- [ ] Session-level audit trail tracks all side effects and spend
- [ ] Spend governor is active with per-session limits
- [ ] State checkpoint pattern is implemented so partial failure can resume, not restart
- [ ] Each tool in the chain is tagged with its reversibility class
- [ ] Credential mode logging is active — know which identity each call ran under
The servers that feel mature in production aren't necessarily the most capable. They're the ones that make debugging easy.
Part of a series on production-safe MCP deployments:
