After the Rush to Build: What Carriers Learned About AI for Medical Record Review

This Thought Leadership is part of a series brought to you by SiftMed.

View series

A small team can now build an AI tool that reads medical records in a few weeks. Keeping it accurate on live claims is harder. Many carriers have learned that the work doesn’t end when the first version is built.

The rush started in late 2025, when AI models took a big step up and new tools made it easier for small teams to build their own applications. With adjusters spending 2 to 6 hours a day reading records, many carriers started to build.

By the second half of 2026, the ongoing costs became more apparent: tokens, maintenance, and re-testing every time a model changed. Many are once again weighing build or buy, though most carriers already use some mix of an in-house tool, a specialized platform, and a general AI assistant.

Building a tool is easier than ever, and for carriers with the technical resources to support it, there are real advantages. You control the roadmap and shape the tool around your own forms and workflows.

“When you build, the first version comes fast and the demos look great,” says Amir Tavafi, Chief AI Officer at SiftMed, “You will see 80 or 85 percent accuracy early and it will feel close to done, but the bigger investment is in everything after that, and it’s a recurring cost.”

Four challenges tend to emerge when AI moves from a demo to real medical files:

Each of these challenges show up in the budget, and medical data adds another layer of complexity. Processing scanned records, handling handwritten notes and extracting structured information requires more than a model API. Carriers also need the infrastructure, document-processing tools, testing and engineering resources to keep those workflows running reliably. Those costs need to be part of the initial business case, not added later.

Buying moves most of the day-to-day maintenance to a vendor, including model testing, support for new form types and integrations. A vendor working with many carriers may have encountered edge cases a single carrier hasn’t, though that experience only matters if it’s backed by testing and validation.

In return, you give up some control. Your roadmap depends on the vendor’s, your data is processed outside your own systems, and a change you need joins someone else’s queue.

“Buying doesn’t remove the obligation,” Tavafi says. “What it changes is who does the work.”

Buying doesn’t hand off the regulatory responsibility, but specialized vendors typically validates and monitors its models, re-tests them as they change, and maintains the security controls and certifications that protect PHI. The carrier’s job shifts to overseeing that work: asking for evidence, reviewing results and confirming the vendor meets its standards.

On either path, the work starts with defining the problem and setting clear success metrics. “Summarize the file” can mean a chronology, a file overview, summary notes or a list of facts to flag, and each needs different output.

“Accuracy starts before anything is summarized,” says Holly Hill, CEO and co-founder of SiftMed. “With SiftMed, records are sorted, de-duplicated, categorized and dated before data is summarized and before your team’s eyes land on the file.”

Whether you build or buy, claims professionals should ask how accuracy on your output is measured, and how it is re-tested over time.

General AI assistants work well for drafting emails and letters, summarizing a short report and answering policy or process questions. For the core of medical review, they typically need extra work.

Large scanned files can hit upload limits, and page-level citations to support a reserve or a tribunal aren’t built in. Connecting to claims systems such as Guidewire or OnBase takes development. With consumer tools, organizations may have less control over how PHI is accessed, stored and handled. Enterprise versions address some of this, but teams should know what their tool can and can’t do, especially with the sensitive data in a claim file.

Almost any approach looks good on a clean sample file. A few questions can tell you how it will hold up on a real claim.

The industry often describes the adjuster’s role as “human in the loop.” Another way to frame it is to ask how much of the work to automate, and where the team keeps control.

The goal is to automate as much as possible without creating a bottleneck, while keeping control at the points that matter. A required review at every step slows down every claim. The ability to step in where it counts doesn’t.

Automating tasks is different from automating decisions. Sorting, indexing, extracting facts and drafting the summary can run without a manual step at each stage. Adjusters still need to verify sources, look into exceptions and challenge outputs, and the claim decision stays with them.

Adjusters should be involved in selecting and testing new tools, too. They know where the friction exists in the claims process and can help identify where automation will make the biggest difference.

Getting AI to produce a good summary on day one is the easy part on any path. The decision that matters is who maintains accuracy a year later, with hundreds of new records, form changes, and many new models.

A carrier that builds needs a team ready to own that work for years. A carrier that buys needs a vendor that can prove it does that work.