By: Keith Dutton, Vice President of Engineering, and Anav Sanghvi, Senior Staff Software Engineer
The Model Context Protocol (MCP) is rapidly changing how we think about AI integration, but it’s still very much a moving target.
In our journey of building healthcare AI agents with MCP, we’ve moved past the initial excitement and into the “learning by fire” phase. We quickly learned that success requires more than connecting tools to a model. It requires disciplined context management, strict validation, and carefully controlled data exposure to ensure patient safety, operational accuracy and provider trust.
In the spirit of “cutting our fingers so you don’t have to,” we’ve compiled our hardest-won lessons from the trenches of MCP development.
TL;DR
Here’s what we’ve learned so far from building AI agents with MCP:
- Share only what’s necessary: Limit the data you send to the LLM to avoid unhelpful or unpredictable behavior.
- Segment dense data: Break complex information into discrete, enum-controlled sections to guide the LLM’s choices.
- Use deterministic guardrails: In healthcare, some things just have to be right. Use hard, rules-based checks, not just LLM-based review, to validate outputs before they reach users.
- Maintain server-side session state: Keep a source of truth independent of the LLM and validate every action against it to catch hallucinations.
The Fundamental Shift: MCP is Not a REST API
The most important realization we had is that MCP isn’t just another version of a REST API. In traditional software, APIs return comprehensive, structured data for deterministic code to consume; the calling code knows exactly which fields it needs and ignores the rest. MCP, however, is built for an intelligent interpreter: the LLM.
When the consumer is an LLM, the rules of data exchange change entirely. You aren’t just passing strings; you’re passing reasoning context. Every field you return is something the model might latch onto, interpret, or volunteer to the user. This shift changes everything, from how much information you should expose to how you manage state and sessions.
The below takeaways have allowed us to move from “wonky” AI behavior to agents that actually work in a high-stakes healthcare environment.
1. Don’t Overload Healthcare AI Agents with Information
We discovered early on, much to our amusement, that you can give an AI agent too much information. Initially, we built a tool to fetch patient appointments that provided both a start and an end time. Seemed logical, right?
The problem was that the LLM became “too smart” for its own good. It would calculate the appointment duration and inform patients that their slots were only 10-20 minutes long. In a doctor’s office where over-scheduling is the norm, that’s information providers prefer to keep private to avoid patient confusion or frustration.
The Fix
Our response came in two stages:
- First attempt – a prompt patch: We added an instruction to the system prompt: “Don’t mention the duration.” This reduced how often the agent shared appointment length, but it was fragile. Prompt instructions are suggestions, not guarantees; the LLM would still occasionally slip, especially in longer conversations.
- The real fix – remove the data entirely: We redesigned the tool so it simply doesn’t return the end time anymore. The appointment tool now gives the LLM only what the patient actually needs: the date, start time, provider name, facility, and day of the week — no end time, no duration. If the LLM never sees the data, it can’t leak it.
The lesson: prompt instructions are band-aids. The robust solution is controlling what data the tool exposes in the first place. Less data means fewer opportunities for the model to make unhelpful inferences.
2. Curb “TMI” with Enums and Tool Wrapping
In healthcare, drug data is notoriously dense – a challenge we faced firsthand while developing a tool for medication information retrieval. Working with one of our partners, we found that their MCP server often returned a “wall of text” that consolidated dosage instructions, overdose effects, and side effects into a single, overwhelming block. This flood of information (aka “TMI”) would overload the agent, causing it to struggle to find specific answers, or worse, spit out irrelevant information.
The Fix
We “wrapped” the third-party tool with our own targeted version. Instead of dumping all the data at once, we segmented the information into 8 distinct categories: things like uses, side effects, precautions, drug interactions, and storage. The LLM can only request one category at a time, and only from that predefined list.
So if a patient asks “what are the side effects of my medication?”, the agent fetches just the side effects, not a 2,000-word monograph covering everything about the drug. If the LLM tries to request something outside the predefined list, the tool simply rejects it, making things more stable and predictable.
Moral of the story? Be surgical with the data you send back, and simplify what the LLM gets to avoid unnecessary confusion.
3. Use Deterministic Guardrails (Because Some Things Just Have to Be Right)
There’s a growing ecosystem of guardrail approaches for LLMs, including using a second LLM to review the first one’s output. But in healthcare, some information simply has to be correct, every time, with no room for probabilistic “good enough.” That’s where deterministic guardrails come in: hard, rules-based checks that don’t rely on another model’s judgment.
Dates and times are a perfect example. LLMs don’t inherently understand what “now” means. Their knowledge is based on training data, which can be months or years old. So when a user says, “Schedule an appointment for next week,” the LLM has no idea when “next week” is unless you tell it the current date and time. Even then, mistakes can happen.
For example, even after injecting the current date into the system prompt, the LLM can occasionally get the day of the week wrong, like calling July 10th a Thursday instead of a Wednesday.
Our Solution
Even with the right date injected, the LLM can still slip up, like telling a patient their appointment is on “Thursday, July 10th” when July 10th is actually a Wednesday. So we built a validation layer that scans every response the LLM generates and checks any dates it mentions against the real calendar. If the day of the week doesn’t match the date, the system catches the error and forces the LLM to correct itself before the patient ever sees it. The wrong date never makes it through.
Long-term scheduling, especially in family medicine, adds even more complexity. Managing available slots across months or years without overwhelming the LLM’s context window requires careful planning. The trick is to keep context lean and validate relentlessly.
4. Tackle Hallucinations with Server-Side Session State
While REST APIs tend to be stateless, MCP tools can maintain their own server-side session, a data store that’s independent of both the MCP protocol’s session lifecycle and the LLM’s conversation context. This distinction matters: we’re not relying on the LLM to remember what’s real. We’re keeping a source of truth on the server that the tools validate against.
This is critical for preventing hallucinations during sensitive actions like appointment cancellations. If a patient wants to cancel an appointment, the agent needs to reference the correct one. Without a server-side safety net, an LLM might confidently reference an appointment that doesn’t exist — a hallucination that, unchecked, could reach the real scheduling system.
Here’s how our safety net works:
1. When a user starts a conversation about their appointments, we fetch their upcoming appointments and store them in a server-side session.
2. The user says, “I want to cancel the one on Friday.” The LLM identifies the right appointment from the data it was given.
3. When the LLM calls the cancellation tool, the tool checks the ID against the server-side session: does this match something we actually fetched? If not, the request is rejected – the hallucination never reaches the external system.
We apply this same pattern beyond appointments – for provider lookups, visit reason triage workflow steps, and other sensitive operations. And importantly, the tools don’t just reject bad input with a generic error. They return corrective context, information that helps the LLM understand what went wrong and try again with valid data. This creates a reflective loop: the LLM proposes an action, the tool validates it against the server-side session, and if it fails, the tool guides the LLM toward a correct response rather than just slamming the door.
The takeaway: don’t rely on the LLM to be your source of truth. Maintain your own, validate against it, and when validation fails, help the LLM recover.
Key Takeaways
Building with MCP is a journey of shifting from “giving the computer data” to “guiding the AI’s reasoning.” The patterns we’ve outlined, stripping unnecessary fields, wrapping tools with enums, validating outputs in real-time, and anchoring actions to session state, aren’t theoretical. They’re running in production today, handling real patient interactions.
If there’s one thread that ties these lessons together, it’s this: don’t trust the LLM to do the right thing with extra information…design your tools so the right thing is the only option.
The road isn’t always smooth, but with the right constraints and validation layers, MCP becomes a powerful foundation for healthcare AI that providers can actually trust.
