How to evaluate an AI vendor before you buy

Every demo is built to win. That is its job. The vendor picks the inputs, rehearses the path, and quietly leaves out whatever breaks. Before you wire that product into your workflow and hand it your data, the work is to find what the demo was built to keep out of frame — and with AI, that gap is wider than most buyers expect. A genuinely impressive demo can be assembled in a weekend on top of someone else’s model, with very little underneath that the vendor actually built. You are not trying to catch them out. You are trying to know what you are paying for, and what you are committing your business to, before you sign.

Start with what the vendor actually built

Almost every AI product is a layer on top of a model from OpenAI, Anthropic, or Google, and that is fine, because almost everyone’s is. But it tells you where to look. The model is not really what you are buying — you could reach most of the same raw capability through a chat subscription. What you are paying the vendor for is the layer they put on top: the workflow that turns a general model into something that does your specific job, the integrations into your systems, the reliability, and the support when it breaks.

So ask what that layer actually is. If it is a real product — one that took genuine work to build and would take a competitor genuine work to copy — it can be well worth the price. If it is a thin wrapper around a public model and a clever prompt, you are paying a markup for something you could get more directly, and backing a vendor who can be undercut by the next one. Either answer is useful. You just want to know which one you are signing.

Make them show how they know it works

Think about what these products actually do: answer your phones, draft your team’s email, score the leads coming in, read your documents. Each one is easy to make impressive for the length of a demo. The real question is whether it holds up across the next ten thousand cases — and whether the vendor even knows. So ask for their evidence, not the model maker’s. The benchmark scores OpenAI or Anthropic publish describe the engine; they say nothing about the product this vendor built on top of it. What you want to see is how they tested the thing they are selling you:

• The scenarios they ran it through, especially the ugly ones. The bad phone connection, the strong accent, the angry caller, the document that does not fit the template. A vendor who only ever shows the happy path has not met its own product in the wild — and once you sign, you are the one who will.
• Whether any of it is written down. A serious vendor can show you what they tested, where it failed, and what they changed. A weak one points back at the demo. If the only record of quality lives in a founder’s head, quality is whatever they remember on a good day.
• Whether a wrong answer can be traced. Ask them to pull up a real case that went badly and walk you through what happened — what the system saw, what it did, and why. A vendor who can follow that trail can fix the product and answer for it when it affects your customers. One who can only shrug is handing you that shrug.

This matters more than it sounds. Most AI products that disappoint do not disappoint because the model is weak. A 2024 RAND study traced the leading causes of failed AI projects to the problem being framed wrong, the data not being ready, and infrastructure that could not carry the work. A vendor with no record of how its product was tested is a vendor who will find those cracks on your time.

Then prove it yourself, on your own work

Here is the advantage you have that an investor never does: you can try it before you commit. Take the product away from the sales team and run it on inputs they did not choose — your own documents, your own awkward cases, the messy real examples it will actually meet in your business. Insist on a pilot on your data before you sign anything longer than a short trial. The distance between the rehearsed demo and the same product on your reality is the single most useful measurement you can take.

Many vendors, especially younger ones, will not have rigorous testing of their own to show you, and at that stage that is not automatically a red flag — what is fatal is a vendor with no way to tell whether the product works at all. When their evidence is thin, your pilot becomes the evidence: a couple dozen real cases, run through the product, checked by someone on your side who knows the right answer. A product that holds up is worth paying for. One that only works on the three examples in the deck is a science project you would be funding.

Where your data goes

For most buyers this matters more than anything on the slides. When you use the product, your data goes somewhere — and with AI it often leaves the vendor’s walls entirely, because the product hands it to a model run by OpenAI, Google, or someone else to get an answer. Before you commit, get clear answers to a short list. Where does your data go when you use the product? Is it used to train the vendor’s system, or anyone else’s? Who can see it? How is it secured? And would the whole arrangement survive a review by whoever owns compliance in your business?

If you handle legal, health, or financial information, these are not questions for the sales call — they belong in the contract. Plenty of tools look fine until someone asks where the data actually goes, and by then your clients’ information may already be flowing through three companies you never evaluated.

What it costs you, and what you are locked into

You will not see the vendor’s costs, and you do not need to. What you need to understand is your own exposure. Start with the price you will actually pay: how does it behave when your usage grows — flat, per seat, per call, metered — and is there a cliff or an overage rate waiting at scale? What happens at renewal, once your business depends on the tool and your leverage is gone? Treat a price that looks too cheap as a question rather than a gift — a vendor selling below its own model cost to win the market is quoting you a future increase, or a vendor that may not last.

Then there is lock-in. If you wanted to leave in a year, could you? Can you export your data and your configuration, or would walking away mean starting over? And ask about the part nobody in the room controls: the model underneath is someone else’s product, and if its provider raises prices, retires a version, or changes terms, your vendor either absorbs that or passes it to you. A vendor who has thought about it has a fallback. One who has not is renting its core capability from a company it cannot influence — and so, in the end, are you.

Can you actually rely on it

A demo runs once, with someone watching. Your business needs the product to work every day, unattended, including on the day it gets something wrong. Ask what happens then: does the product catch its own mistakes and hand off to a person, or does a wrong answer go straight to your customer? What is the uptime, what does support actually cover, and who do you call when it stops working in the middle of a workday? For anything you are putting in front of clients or relying on to run operations, get the service commitments in writing. The quietest failure in AI is a vendor that can demo an impressive thing once and cannot keep it standing in your hands.

The buy test

By the time you are ready to sign, you should be able to answer a short list of plain questions:

• What did the vendor actually build, and what are they just renting from a model provider?
• How do they know it works, and did it survive contact with your own data and your messiest cases?
• Where does your data go, and would that survive a compliance review?
• What does the price do at scale and at renewal, and how hard is it to leave?
• What breaks if the vendor’s model provider changes the price, the version, or the terms?
• Can you rely on it day to day, and who answers when it is wrong?

If the answers are clear and they hold up, you can buy with confidence. If they are vague, the demo was the product, and you should treat the contract accordingly.

None of this is about assuming the vendor is hiding something. It is about knowing what you are committing your business to before the demo becomes a dependency. The good news is that as a buyer you hold leverage an investor never does: you can pilot the product on your own data, and you can put the answers you need into the contract instead of taking them on faith. Done well, this kind of evaluation tells you what is real, what is fragile, and what is missing — in time to negotiate it, or to walk. For a single scoped buy-or-build call, an AI Decision Memo is the closer fit, and the AI roadmap piece covers the build-versus-buy decision from the inside.