All posts
Vivek Vaidya Updated llmobservabilitycost-trackingopen-source

Your AI Stack Needs a Control Plane

You trace every database query. You monitor every microservice. You have dashboards for API latency, error rates, and throughput.

Your LLM calls — the ones that cost a dollar each — run in total darkness.

model = "claude-sonnet-4-6"

Hardcoded in 30 places across your codebase, applied to every task regardless of complexity. No attribution to features or users. No visibility into what’s working and what’s wasteful. No safety net when a provider has a bad day.

Classification that returns one of five labels? Sonnet. Extracting a date from an email? Sonnet. Summarizing a paragraph? Sonnet.

It works. But without a control plane, you can’t see, manage, or optimize any of it.

The Gaps a Dashboard Won’t Close

Plenty of tools will show you aggregate LLM spend. That’s useful, but it’s not actionable. Knowing you spent $2,400 on Anthropic last month doesn’t tell you what to do differently. And it doesn’t solve the operational problems that come with running AI in production:

The invoice nobody can explain. Your AI spend is one of the fastest-growing line items in the P&L — and nobody knows which team, feature, or experiment is driving it.

Three teams, zero shared data. Engineering wants the flagship model for quality. Finance wants costs cut 40%. Product wants three new features shipped. Without data, everyone is guessing.

Single provider, single point of failure. One API outage, one rate limit spike, and your application stops working — with no safety net and no recourse.

What you actually need is the ability to answer questions like:

  • “For my document classification feature, does Haiku produce the same labels as Sonnet?”
  • “If I switch my extraction pipeline to a cheaper model, do the structured outputs still validate?”
  • “Which of my features are latency-sensitive enough that a faster, cheaper model would actually improve the user experience?”

To answer these questions, you need a control plane — not just metrics, but the operational infrastructure to observe, route, and optimize every LLM call.

Introducing Majordomo: The Control Plane for Your AI Stack

Majordomo gives every LLM call a control plane — cost visibility, routing, replay, and evals — in two layers you can adopt however you like.

The platform is a transparent gateway plus a dashboard. Point your traffic at it and you get an Observe view (cost analytics, per-request logs, agent-run cost waterfalls) and an Optimize view (replay, evals, and live experiments). Run it fully managed at gateway.gomajordomo.com, or self-host the same platform inside your own VPC.

The open-source building blocks are there when you’d rather assemble it yourself: majordomo-gateway, a standalone Go proxy that prices and logs every request to your own Postgres with nothing phoning home, and majordomo-llm, a Python client with cost tracking and provider cascade built in. Both MIT-licensed.

Start managed in minutes, self-host the platform, or wire in the open-source pieces — it’s the same control plane either way.

The platform: gateway + dashboard

At the center is a transparent proxy. Point your LLM API calls at it instead of the provider, add one header, and every request is priced and logged. No SDK changes, no code refactoring.

# Before: direct to Anthropic
curl https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -d '{"model": "claude-sonnet-4-6", "messages": [...]}'

# After: through the control plane (one header added)
curl https://gateway.gomajordomo.com/v1/messages \
  -H "Authorization: Bearer $ANTHROPIC_API_KEY" \
  -H "X-Majordomo-Key: $MAJORDOMO_KEY" \
  -H "X-Majordomo-Feature: document-classification" \
  -d '{"model": "claude-sonnet-4-6", "messages": [...]}'

What you get from the gateway:

  • Automatic cost calculation with real-time pricing (refreshed hourly from llm-prices.com, so you’re never working with stale numbers)
  • Full request/response body logging to object storage you own — the payloads you’ll need for replay and optimization, written to your S3 or GCS bucket while the control plane keeps only metadata
  • Custom metadata via headers — tag requests with feature name, user ID, session, workflow step, or anything else, then query your costs by any dimension that matters to your team
  • Multi-provider support — OpenAI, Anthropic, Gemini, Bedrock, and OpenAI-compatible providers, with automatic provider detection from the request path
  • Proxy keys for multi-tenant setups — give each customer or team their own key, map it to real provider credentials (encrypted at rest with AES-256), and track usage independently

On top of the proxy sits the dashboard: cost analytics sliced by model, provider, and metadata; a searchable request log; and agent-run waterfalls that roll a multi-call agent or conversation up into a single trace with a total cost. Run all of it managed, or deploy the platform into your own VPC — the dashboard works identically, and your prompts never leave your network.

The open-source building blocks

Prefer to build it yourself? Two MIT-licensed pieces stand on their own.

majordomo-llm is a Python async client for when you want the control plane built directly into your application code. Every response carries token counts, cost, and latency as part of the object:

from majordomo_llm import get_llm_instance

llm = get_llm_instance("anthropic", "claude-sonnet-4-6")
response = await llm.get_response("Classify this document: ...")

print(response.content)          # "Category: Legal"
print(response.total_cost)       # 0.000842
print(response.response_time)    # 1.23
print(response.output_tokens)    # 12

The library also gives you:

  • Structured outputs that work consistently across providers. Pass a Pydantic model, get a validated instance back — whether you’re using OpenAI (JSON Schema mode), Anthropic (tool calling), or Gemini (response schema). One interface, provider-specific optimizations under the hood.
  • Cascade failover — define a priority list of (provider, model) pairs with LLMCascade. If your primary provider errors, the client automatically tries the next one. Combined with per-provider retries, that’s the safety net for the single-point-of-failure problem — no application code changes required.
  • Async request logging — fire-and-forget logging to PostgreSQL, MySQL, or SQLite, with request/response bodies stored in S3 or the filesystem. Zero impact on request latency.

majordomo-gateway is the standalone, open-source proxy: a compiled Go binary you git clone, build, and run against your own Postgres. It relays your own provider key upstream (it never stores credentials), prices and logs every request, and exposes usage through a CLI and an MCP server so you can ask Claude Code what your agents actually cost. No UI to run, and nothing phones home. It’s the self-contained option when you don’t want the hosted platform at all.

Works With Your Stack, Not Instead of It

A control plane sits underneath your application — it doesn’t replace it.

If you’re building with Pydantic AI, Agno, or any other framework, you don’t need to change anything fundamental. The gateway is a transparent proxy — your framework talks to the gateway instead of directly to the provider, and everything else stays the same. Point the framework’s provider client at the gateway base URL, add the X-Majordomo-Key header, and you’re done — no extra dependency required.

This is a deliberate design choice. Pydantic AI and Agno are excellent frameworks for building AI agents — tools, conversation management, structured outputs. They’re not trying to be operational infrastructure, and they shouldn’t have to be. Majordomo handles the control plane so your framework can focus on what it does best.

From Observability to Optimization

This is where the control plane pays off. Visibility is step one. Optimization is the goal.

Say you’ve been running your document classification feature on Claude Sonnet for three months. The platform has logged every request and response. Now you want to know: could Haiku handle this?

That’s a replay. In the dashboard you pick the traffic to test, choose a challenger model, and optionally add an LLM judge to score equivalence:

source:
  filters:
    feature: document-classification
  model: claude-sonnet-4-6
  days: 30
  limit: 50

target:
  provider: anthropic
  model: claude-haiku-4-5

judge:
  enabled: true
  provider: openai
  model: gpt-5-mini

The platform then:

  1. Pulls the logged requests — filtered by metadata, model, and time range
  2. Replays each prompt against your challenger model
  3. Compares outputs — exact match first, then the LLM judge for semantic equivalence
  4. Reports back — cost and latency comparisons, match rates, and the specific examples where the models diverged

If Haiku agrees with Sonnet 98% of the time on this task, you switch and save around 90% on that feature’s LLM costs while getting faster responses. If it doesn’t, you know exactly where and why.

You’re not guessing based on benchmarks. You’re testing with your actual production data, your actual prompts, your actual edge cases. That’s the difference between a dashboard and a control plane.

Replay is the start of the Optimize story, not the end of it:

  • Evals turn those judgments into a reusable scorecard. Build an eval set from logged requests, define your scoring criteria, and run it against any model to get an aggregate quality score before you ship.
  • Experiments take it live. Split real traffic across model variants by weight, watch per-arm cost, latency, and error rates in real time, and promote the winner with sticky assignment keeping each user on a consistent variant — all without touching application code.

Where Majordomo Fits

There are great tools in this space. LiteLLM, for instance, offers a comprehensive platform with support for 100+ providers, virtual keys, budgets, and rate limiting. If you want an all-in-one solution, it’s a solid choice.

Majordomo takes a different approach: a modular control plane you adopt on your terms.

  • Want it turnkey? Managed cloud — sign up, point your SDK at the gateway, and watch cost and usage roll in.
  • Need data residency? Self-host the platform in your own VPC. Same dashboard, same features, prompts that never leave your network.
  • Already have a framework? The gateway works as a transparent proxy underneath Pydantic AI, Agno, or anything else that makes HTTP calls.
  • Want just the library? Use majordomo-llm standalone with its built-in cost tracking and cascade. No gateway needed.
  • Want a free, self-contained gateway? Run the open-source majordomo-gateway — no dashboard, no dependencies, nothing phoning home.

Use whichever pieces make sense. Leave the rest.

Get Started

The fastest way to start is Majordomo Cloud — sign up, point your SDK at the gateway, and watch cost and usage roll in. Prefer to run it yourself? The gateway and client library are open source under the MIT license.