For a few months now, every essay here has ended with a single question to ask an AI vendor. Readers kept asking for them in one place — ideally something they could keep open during a pitch. So here they are: five questions that, between them, separate AI you can actually trust in serious work from AI that just demos well.
None of them is technical. You don't need to understand how the model works to ask any of these — you only need to watch how the person across the table answers. A good vendor welcomes them. A nervous one reaches for buzzwords. That tell alone is worth the meeting.
1. Where does our data actually go?
The very first thing you hand an AI tool is information — often your clients'. So get specific before anything else: where is our data stored, who can access it, does any of it leave our environment, and will you put that in writing? A plain, concrete answer is a good sign. A reassuring "we don't train on your data" followed by a change of subject is not. More on this →
2. Where do its answers come from — general training, or our data?
A tool running purely on general knowledge is a very smart stranger to your business. Ask whether it's grounded in your own documents and policies, or just guessing from what it read on the open internet. That difference decides whether an answer is merely right in general or actually right for you. More on this →
3. What does it do when it doesn't know?
The dangerous answer is never the obviously wrong one — it's the confident one that happens to be wrong. Ask them to show you the tool at its edge: does it guess to fill the gap, or does it flag uncertainty and hand off to a human? An assistant that always has an answer is a liability dressed up as a feature. More on this →
4. Where will this live in our team's day?
The smartest tool nobody opens is worthless. Find out whether it works inside the tools your people already use or sits off in its own tab — and whether using it removes steps or adds them. Most AI doesn't fail on intelligence; it fails on adoption, and adoption is decided by fit. More on this →
5. When it's wrong, who's accountable — and can you prove why it decided?
Every question above assumes something will eventually go wrong, because it will. So ask who is responsible when it does, and whether they can show the real basis for a decision — the sources and factors behind it — not a tidy story generated after the fact. In regulated work, an answer you can't trace is an answer you can't defend. More on this →
Keep This Handy
Screenshot this, or paste it into the notes for your next vendor call:
- Where does our data go — who can see it, and will you put it in writing?
- Do answers come from our information, or just general training?
- What does it do when it doesn't know — guess, or flag and hand off?
- Where does it live in our day — and does it remove steps or add them?
- When it's wrong, who's accountable, and can you show why it decided?
These aren't gotchas. They're the same questions we hold ourselves to at EMDELLE — the bar we think any AI touching serious work should clear, which is exactly what we built BAG, our behavioral engine, to meet. (One mention, as always — this post is here to be useful, not to pitch.)
Ask all five the next time someone sells you AI. The demo will tell you what the tool can do on its best day. These five will tell you whether you can trust it on an ordinary one.
See you next Friday — one more thing to learn.
— Tia Lake, Founder & First Steward