Hospitals want automation without autonomy, and health AI startups should pay attention 

When Becker’s Hospital Review asked hospital supply chain leaders in late October how they would spend an extra $1 million USD, most picked technology that takes manual work off their teams, from RFID tags and electronic shelf labels to warehouse robots. A week earlier, a different group of leaders told the same publication that autonomous AI is one of the most overhyped trends in their field. 

Read side by side, the two pieces describe a buyer that is desperate for help and wary of software that decides things on its own.

That tension matters for any startup building AI for healthcare operations. Hospitals are clearly ready to pay for tools that remove repetitive coordination work, but are much less ready to hand over judgment. 

The companies most likely to earn their trust will be the ones that design for that from day one, deciding in advance which calls the software makes alone and which ones it sends to a person.

In September, the VA Office of Inspector General reported that the main operating room at the VA Augusta Health Care System in Georgia closed four times in five months, once for 27 days, because of incomplete surgical kits, expired supplies and unreliable restocking. The health system had already awarded a $3.2 million contract for an automated inventory platform. It was never fully installed, a former supply chief disabled it in 2023, and inspectors found some of its equipment abandoned in a warehouse.

What hospital buyers are saying

The skepticism in the October Becker’s piece was specific: Joe Wilson, chief procurement officer at Montefiore Health System, argued that improving data integrity, contract compliance and clinical standardization often delivers faster, more reliable returns than a new AI forecasting project, and that organizations with strong fundamentals will be better placed to use AI later. 

Michael Alfaro of Community Memorial Healthcare said the bigger problem is the expectation that a universal, out-of-the-box predictive tool is around the corner, when the successful models he has seen are heavily customized to each hospital’s systems, whereas Nalin Kulasekara of Hospital Sisters Health System took aim at dashboards. 

“Having visibility into everything doesn’t solve problems,” he said.

Neither leader rejected AI outright, and notably so. Earlier this year, supply chain executives told Becker’s that the technology’s most useful applications so far have been eliminating repetitive work, improving visibility into operations and helping staff find information faster, rather than replacing procurement teams or making purchasing decisions autonomously.

Large health systems are buying along those lines. Mount Sinai Health System said in spring it would deploy Clarium’s supply capture and preference card technology across eight hospitals and surgery centers, and Indianapolis-based Community Health Network is using the same company’s platform to cut surgical supply costs across nine hospitals. Medline, the distributor, began piloting its own AI platform with 10 U.S. health systems in February.

Designing for oversight instead of bolting it on

For smaller companies, the question is how to build something that fits that buyer. Adrián Figueroa, an electrical engineer and Stanford Graduate School of Business graduate who founded the healthcare logistics firm Flexio and later Routina, a platform for coordinating information between medical distributors and healthcare organizations, has built his answer around a limit he sees in human review itself.

“With AI as it stands now, the volume of output is more than a human can process,” he said. “Reviewing everything turns the human into the bottleneck, but human judgment is still needed for high-risk or low-confidence decisions.”

His approach is to have the system score each output for confidence and for risk, and route only the uncertain or consequential ones to a person, while tracking how well it performs on everything it handles alone. 

The design only works, he argued, if the reviewers have real authority. “Human oversight has to be real, not a person clicking ‘approve’ all day,” he said, adding that specific people should own flagged decisions, be able to override the system and have every override logged and reviewed.

That lines up closely with where regulators are heading. Article 14 of the EU AI Act requires high-risk AI systems to allow effective human oversight, including the ability to override them, and the U.S. Department of Health and Human Services’ AI strategy, released in December 2025, asks its divisions to apply minimum risk management practices, human oversight among them, to high-impact systems. 

For a startup selling into hospitals or health agencies, oversight features are increasingly something buyers will ask about before they sign.

Figueroa also warned about the data feeding these systems. “If the input data is biased, AI will apply that bias faster and at scale,” he said, which is why he believes teams need to keep reviewing what their models measure and why.

What buyers will demand next

The demand for help is unlikely to ease. Hospital supply chain costs are projected to rise 3.39% in 2027, and the FDA’s medical device shortage list, most recently updated on October 26, still includes products such as neurosurgical patties and sponges. 

Administrative work was already a heavy burden before any of this, with McKinsey estimating that administrative spending accounts for about $1 trillion, or 25%, of total U.S. healthcare spending.

That mix of tight margins and costly manual work will shape what hospitals buy; health systems have little patience left for pilots that never reach the floor, as Augusta’s abandoned inventory system showed, and supply chain leaders are already telling the trade press they want their data and contract compliance fixed before they trust a forecasting model. 

For startups, that points to a narrower but more durable opening: the tools most likely to win are the ones that slot into the coordination work hospitals and distributors already do, prove themselves on routine decisions, and make it obvious who signs off on everything else.

The audit trail may end up mattering as much as the algorithm. As Europe enforces its oversight rules and U.S. agencies formalize their own risk controls, buyers will increasingly expect vendors to show which decisions a system made alone, which ones a person reviewed, and what happened when someone overrode it. 

Companies that build those records into the core of the product will have an easier time getting through procurement, and a stronger position when something goes wrong and a hospital wants to know why.

The way into the market may not run through the hospital at all, though. Most of this year’s headline deployments, at Mount Sinai and Community Health Network among others, involve tools sold directly to large health systems. 

Routina is testing a different route. Surgical Solutions International, which recently launched a new line of medical products in the Dominican Republic with Cardinal Health, plans to manage those products through the platform, and Figueroa said the company intends to pitch Cardinal Health on using it to coordinate the distributor’s partners in other countries. 

Whether or not that pitch lands, it reflects a bet more founders are likely to make. The firms moving products between manufacturers and hospitals feel the coordination problem just as sharply, and they may be quicker to pay for a fix.

Featured image: Planet Volumes via Unsplash+