AI Vendor Due Diligence: 10 Questions to Ask Before You Sign

Now comes the hard part: figuring out if any of it is true.

For mid-market operations leaders, evaluating AI vendors has become one of the most frustrating parts of the job. New tools launch every week. Every vendor claims to be "AI-powered." And most of your team doesn't have time to become AI experts on top of everything else they're doing.

Here's the uncomfortable reality: 95% of AI pilots fail to deliver business impact. And 40% of companies calling themselves "AI-powered" use virtually no AI at all.

This isn't a guide about finding the "best" AI vendor. It's about protecting your budget, your data, and your team's time from vendors who overpromise and underdeliver.

The Vendor Overwhelm Is Real

If you feel like you can't keep up with AI vendors, you're not alone.

BCG researchers put it bluntly: "No CIO, CTO, or CDO can manually keep up with the flood of new products, wrappers, and platforms that appear each month."

The result? Buyer fatigue. According to NPI Research, software vendors have become procurement's "biggest pain point." The top complaints are licensing complexity and lack of vendor responsiveness.

Meanwhile, 82% of enterprises are actively trying to reduce their vendor count. But the AI gold rush keeps adding more options to evaluate.

For mid-market teams without dedicated procurement staff, this creates a real problem. You don't have time to evaluate dozens of vendors. So you need to know exactly what questions matter—and what red flags to watch for.

AI Washing: The 40% Problem

Let's start with the most basic question: Is this vendor actually using AI?

It sounds ridiculous. But research from MMC Ventures found that 40% of European tech firms calling themselves "AI startups" used virtually no AI at all. A 2025 survey of 1,200 fintech companies found the same thing—40% of "AI-first" companies had zero machine learning code in production.

This isn't a gray area. Regulators are taking notice. The FTC has filed twelve AI-washing cases since 2024. The SEC settled with two investment advisors for $400,000 each after they "marketed to clients that they were using AI in certain ways when, in fact, they were not."

Red flags that suggest AI washing:

• Vague "AI-powered" claims without explaining how the AI works
• No evidence to support claims of industry-leading performance
• Promises that AI can "do everything"
• Resistance to providing technical explanations
• AI that doesn't improve over time (real AI learns and adapts)
  ‍

What to ask instead:

"What specific AI model type powers this feature? Can you explain the trade-offs you made when selecting it?"

A trustworthy vendor will answer this clearly. A vendor hiding behind buzzwords will dodge the question.

The Hidden Cost Problem

Here's a stat that should make every operations leader pause: Enterprise AI implementations typically cost 3-5x the advertised subscription price.

That's not a typo. Research from USM Systems found that hidden costs can inflate total ownership by 200-400% compared to initial vendor quotes.

One fintech startup budgeted for an AI sales agent. Five months and $72,000 later—triple the initial quote—they finally got it working. The culprits? Unanticipated API licensing fees and CRM integration issues nobody mentioned during the sales process.

Where the hidden costs hide:

Data preparation (50-70% of your project budget): Before AI can work, your data needs cleaning, labeling, and structuring. Most vendors assume this is already done. It rarely is.

Integration complexity: Connecting AI tools to your existing CRM, ERP, or operations systems can increase costs by 40-60%. Legacy systems make this worse.

Infrastructure scaling: If you're processing significant data volumes, GPU and cloud costs can reach $25,000+ monthly.

Training your team: Budget 10-15% of your implementation cost for workforce development. AI tools are useless if nobody knows how to use them.

Compliance and security: GDPR, HIPAA, and other compliance requirements can double your implementation budget.

Questions to ask:

• "What does total cost of ownership look like for a company our size? Include integration, training, and infrastructure."
• "What pricing model do you use? Are there variable costs based on usage?"
• "What hidden fees have your other mid-market customers encountered?"
  ‍

If the vendor can't answer these questions with specific numbers, that's a red flag.

The Data Rights Problem Nobody Talks About

This is the sleeper issue that catches most buyers off guard.

According to TermScout analysis, 92% of AI vendors claim broad data usage rights in their contracts. That's far higher than the 63% average for typical SaaS agreements.

What does "broad data usage rights" mean in practice? It often means the vendor can use your inputs—your customer data, your prompts, your operational information—to train their AI models. Models that serve your competitors.

Recent lawsuits make this risk concrete. Figma faces accusations of using customer design files for AI training without consent. Salesforce is being sued over allegedly training models on datasets from potentially pirated material.

The legal landscape is shifting fast. But right now, most legacy SaaS contracts were never written with "model training" in mind. That gap can cost you.

What to negotiate:

• Explicit prohibition on using your data for AI model training
• Clear ownership of any AI-generated outputs
• Opt-out rights if you later decide you don't want your data used
• Specific language about what happens to your data after the contract ends
  ‍

The question that reveals everything:

"Will any of our inputs, prompts, or usage data be used to train your models? If so, how can we opt out?"

Watch how the vendor responds. Hesitation or vague answers are warning signs. A trustworthy partner will answer directly and offer clear contractual protections.

The 10 Questions Framework

Based on guidance from AI governance experts, here are the ten questions every mid-market operations leader should ask before signing an AI vendor contract:

Technical Reality

1. What AI model type powers this solution? Each model type has different trade-offs. You don't need to understand the technical details—but the vendor should be able to explain why they chose their approach.

2. What data was used to train this model? Every training dataset has biases and limitations. Understanding the data source helps you understand what the AI will be good at—and where it might fail.

3. Will our inputs be used as training data? This is the data rights question. Get a clear yes or no, and get it in writing.

Performance and Limitations

4. What evaluations and testing have been done? Quality vendors put their AI through rigorous testing, including "red-teaming" where experts try to break it. Ask for documentation.

5. What are the known limitations? Every AI system has specific things it can't do well. A vendor who claims there are no limitations is either lying or doesn't understand their own product.

6. How do you detect and manage model drift? AI performance changes over time as data patterns shift. Ask how they monitor this and what happens when performance degrades.

Security and Compliance

7. What security frameworks are you certified against? Look for SOC 2 Type II and ISO 27001 at minimum. Ask to see actual certificates, not just claims.

8. What bias metrics do you use? "Fair and unbiased" means nothing without quantification. Ask for specific metrics and testing results.

Operations and Support

9. How is model behavior monitored in production? Things go wrong. Ask how they detect issues and how quickly they respond.

10. What happens to our data after the relationship ends? Some contracts allow vendors to keep using your data even after you cancel. This should be explicitly addressed.

The Reference Check That Actually Matters

Most vendors will happily provide references. The trick is asking the right questions to the right companies.

Don't just ask for references. Ask for references from companies similar to yours.

A vendor might have great success with enterprise clients but struggle with mid-market implementation complexity. Ask specifically for customers with similar team sizes, industries, and use cases.

Questions for reference calls:

• "What surprised you about the implementation? What costs or challenges weren't discussed upfront?"
• "How does the vendor handle problems? Can you give me a specific example?"
• "If you were starting over, what would you negotiate differently?"
• "Is the AI actually delivering measurable results? What are they?"
  ‍

The honest answers often come when you ask about what went wrong—not what went right.

Contract Red Flags

Before you sign anything, watch for these dangerous contract terms:

"As-is" warranties on outputs. One vendor's contract stated "outputs provided 'as-is' with no warranty of accuracy." That means if the AI gives your team bad information, you have no recourse.

Vague language about AI features. Terms like "AI-enabled analytics" create disputes later when you try to activate features the vendor claims require additional licensing.

Broad post-termination data rights. Some contracts allow vendors to keep using your confidential information for training purposes after you cancel. Read the fine print.

Unlimited liability caps. If the vendor won't accept reasonable liability allocations, consider it a major risk factor in your selection process.

Automatic renewal with price increases. Standard in SaaS, but particularly dangerous with AI tools where costs can escalate unpredictably.

What to negotiate:

• Data usage limitations that prohibit training on your data
• Clear ownership of any AI-generated outputs
• Performance warranties with specific, measurable thresholds
• Model portability rights if you need to switch vendors
• Explicit compliance commitments with applicable laws

The Vendor Willingness Test

Here's the simplest way to evaluate an AI vendor: Pay attention to how they respond to your questions.

A trustworthy partner will understand that due diligence is standard. They'll provide clear, direct answers. They'll offer documentation without pushback.

A vendor who is hesitant, evasive, or refuses to provide clear answers is telling you something important. Either they lack mature governance processes, or they're not confident their practices would survive scrutiny.

Neither is a vendor you want handling your operations data.

The Bottom Line

Evaluating AI vendors shouldn't require a PhD in machine learning. But it does require asking better questions than most vendors expect.

The good news? Purchasing AI from specialized vendors succeeds 67% of the time, compared to just 33% for internal builds. When you find the right partner, AI can deliver real results.

The key is protecting yourself from the vendors who are all promise and no substance. That means looking past the demos, asking hard questions about costs and data rights, and talking to customers who look like you.

Your operations team deserves tools that actually work. These questions help you find them.

Want Help?

The AI Ops Lab helps operations managers identify and capture high-value AI opportunities. Through process mapping, value analysis, and solution design, you'll discover efficiency gains worth $100,000 or more annually.

Apply now to see if you qualify for a one-hour session, where we'll help you map your workflows, calculate the value of automation, and visualize your AI-enabled operations. Limited spots available. Want to catch up on earlier issues? Explore our resource Hub.

Magnetiz.ai is your AI consultancy. We work with you to develop AI strategies that improve efficiency and deliver a competitive edge.