AI Agent Latency Budget Calculator
Estimate end-to-end agent response time across providers, architectures, and deployment models.
Workload & Architecture
Estimates are directional. Last updated: 2026-07-03. See notes.
Fastest option under your target
Enter workload to see fastest provider.
—
| Provider | Category | P50 (ms) | P95 (ms) | Time to full response | Status | Notes |
|---|---|---|---|---|---|---|
| OpenAI GPT-4o | Frontier API | — | — | — | — | Fast TTFT; good throughput under light load. |
| Anthropic Claude 3.5 Sonnet | Frontier API | — | — | — | — | Slightly higher TTFT; strong for long-context reasoning. |
| Google Gemini 1.5 Pro | Frontier API | — | — | — | — | Competitive throughput; long-context TTFT scales with input. |
| Groq Llama 3.1 70B | Fast inference | — | — | — | — | Very low TBT; excellent for streaming UX. |
| Together AI Llama 3.1 70B | Open model API | — | — | — | — | Cost-efficient; latency varies by load. |
| Local RTX 4090 (Q4_K_M 70B) | Local GPU | — | — | — | — | No network RTT; TTFT limited by prompt processing. |
| Local RTX 3090 (Q4_K_M 70B) | Local GPU | — | — | — | — | Slower prompt processing and decode than 4090. |
| Vast.ai RTX 4090 | GPU cloud rental | — | — | — | — | Add 30-100ms network RTT depending on data center. |
| RunPod RTX 4090 | GPU cloud rental | — | — | — | — | Cold-start can add seconds if not pre-warmed. |
Frequently asked questions
What does TTFT mean?
Time To First Token (TTFT) is the delay from sending a request to receiving the first generated token. It is dominated by prompt processing and network round-trip time.
What is TBT?
Time Between Tokens (TBT) is the average delay between each generated output token. Lower TBT means smoother streaming and faster total response times.
How does retrieval affect latency?
Each retrieval step (vector search, rerank, document fetch) adds its own latency before the LLM can start generating. The calculator adds retrieval time before TTFT.
How do tool calls affect latency?
A tool-calling agent typically emits a tool request, waits for the tool to execute, then generates the final answer. The calculator models that round-trip once per tool call.
Why is local hardware slower on TTFT?
Local GPUs have no network latency, but prompt processing throughput on a single consumer GPU is lower than frontier API clusters. The trade-off is full control and lower per-token cost.
How should I use the P50 and P95 targets?
Set P50 to your desired typical user experience and P95 to the worst acceptable case. The calculator flags when a provider or architecture likely breaches either target.
Latency estimates are directional and assume HTTP/TCP overhead, no queuing, and ideal network conditions. Add 20-40% headroom for production variance.
🤖 Use this tool in your agent
✓ Agent-ready codeCopy the snippet below into your agent, newsletter, or script. The tool page at hermesdispatch.dev/tools/agent-latency-budget-calculator/ is the canonical contract: inputs, outputs, and formulas.
# Hermes Dispatch Tool — AI Agent Latency Budget Calculator
# Source: https://hermesdispatch.dev/tools/agent-latency-budget-calculator/
# Description: Estimate end-to-end AI agent response latency across providers, architectures, retrieval, and tool calls.
# License: MIT (generated by hermesdispatch.dev)
#
# INSTALL:
# 1. Save this file as ~/.hermes/hermes-agent/tools/agent_latency_budget_calculator.py
# 2. Restart Hermes or run /reset in a session
# 3. The tool auto-registers if Hermes uses auto-discovery of tools/*.py
#
# MANUAL REGISTRY (if auto-discovery is off):
# from tools.agent_latency_budget_calculator import register
# register()
import json
DATA = {}
def _ok(result):
return json.dumps({"success": True, "data": result}, indent=2)
def _err(message):
return json.dumps({"success": False, "error": message}, indent=2)
TOOL_NAME = "agent_latency_budget_calculator"
TOOLSET = "agents"
SCHEMA = {
"type": "function",
"function": {
"name": "agent_latency_budget_calculator",
"description": "Estimate end-to-end AI agent response latency across providers, architectures, retrieval, and tool calls.",
"parameters": {
"type": "object",
"properties": {
"ttft_ms": {
"type": "number",
"description": "Time to first token in milliseconds."
},
"tbt_ms": {
"type": "number",
"description": "Time between tokens in milliseconds."
},
"output_tokens": {
"type": "integer",
"description": "Expected output tokens."
},
"retrieval_ms": {
"type": "number",
"description": "Retrieval step latency in milliseconds."
},
"tool_calls": {
"type": "integer",
"description": "Number of tool call round trips."
},
"tool_latency_ms": {
"type": "number",
"description": "Tool execution latency in milliseconds."
},
"network_rtt_ms": {
"type": "number",
"description": "Network round-trip time in milliseconds."
}
},
"required": [
"ttft_ms",
"tbt_ms",
"output_tokens"
]
}
}
}
def _run(args):
ttft = float(args.get("ttft_ms", 0))
tbt = float(args.get("tbt_ms", 0))
output_tokens = int(args.get("output_tokens", 0))
retrieval = float(args.get("retrieval_ms", 0))
tool_calls = int(args.get("tool_calls", 0))
tool_latency = float(args.get("tool_latency_ms", 0))
rtt = float(args.get("network_rtt_ms", 50))
generation_time = tbt * output_tokens
tool_time = tool_calls * (tool_latency + 2 * rtt)
total_ms = ttft + retrieval + generation_time + tool_time + rtt
return _ok({
"total_latency_ms": round(total_ms, 1),
"generation_time_ms": round(generation_time, 1),
"retrieval_time_ms": round(retrieval, 1),
"tool_time_ms": round(tool_time, 1),
"network_rtt_ms": round(rtt, 1),
"note": "P50 target is typical; add 20-50% for P95."
})
def HANDLER(args):
try:
return _run(args)
except Exception as e:
return _err(str(e))
def register():
"""Manual registry hook. Import and call this to register with Hermes."""
try:
from tools.registry import registry
registry.register(
name=TOOL_NAME,
toolset=TOOLSET,
schema=SCHEMA,
handler=HANDLER,
)
except ImportError:
print("Hermes registry not found; skipping manual registration.")
if __name__ == "__main__":
# CLI smoke test
print(HANDLER({}))
Want early access to the next locked tool? Subscribe to The Hermes Dispatch.