Buyer guide
How to Evaluate an AI Engineering Partner: 12 Questions to Ask Before You Sign
Why the usual criteria are not enough
Portfolios, logos and model benchmarks are easy to compare and say little about whether a partner can make AI work inside your systems. The partners who succeed in manufacturing and healthtech tend to share habits: they start narrow, they make sources and uncertainty explicit and they leave you with assets you can maintain. The questions below test for those habits.
Scope and fit
1. Which single workflow would you start with, and why? A good answer names a decision, an owner and a measurable outcome. A vague answer about "AI transformation" is a warning sign.
2. What will you need from us in the first two weeks? Expect a short, specific list: an owner's time, read-only access or exports, a redacted sample, examples of hard cases. Be cautious if production credentials appear on the list.
3. Have you worked with systems like ours? Ask about MES, ERP, LIMS, EHR exports, FHIR or HL7 feeds, or research databases, as relevant. Ask what went wrong last time and how they handled it.
Evidence and evaluation
4. How will each answer show its sources? You want to see records, document versions and rule versions behind an answer, not just a confident paragraph.
5. What happens when data is missing, stale or conflicting? The system should say so explicitly. Ask to see it in a demo. If every demo case has perfect data, ask why.
6. How will we evaluate the pilot? Look for agreed cases with known answers, including difficult ones, and criteria for stopping or continuing that are set before the build starts.
Ownership and maintainability
7. Where will business rules and domain knowledge live? In versioned, readable structures your team can review, not inside prompts or a proprietary platform you cannot export.
8. What do we keep if we stop after discovery or after the pilot? Every deliverable should be usable by your team or another vendor.
9. Can we change the language model later? A good architecture keeps the knowledge layer, rules and evaluation cases independent of any single model, including self-hosted options where data must stay on site.
Risk, data and responsibility
10. What will the AI be allowed to do, and what stays with people? Reading, suggesting, drafting and executing are different permissions. In manufacturing, control and safety systems should stay outside the AI's authority; in healthcare, clinical judgment stays with qualified people.
11. How do you handle confidential and sensitive data? Expect an NDA, synthetic or de-identified data for early work, least-privilege access and a clear statement of where data is processed.
Working together
12. Who will actually do the work, and how will we communicate? Ask to meet the person who will design the system. Agree on time-zone overlap, written updates and how decisions are recorded.
A simple scoring approach
Score each answer from 0 to 2: 0 for vague, 1 for plausible but generic, 2 for specific with an example. Weight questions 4, 5, 6 and 8 double; they predict whether the result will be trustworthy and maintainable.
We are happy to answer these twelve questions about our own work. You can see two working demonstrations first, or describe your workflow and ask us directly.
Questions & answers
Should we choose a large consultancy or a small specialist team?
It depends on scope. Large firms bring breadth and capacity for programs; small specialist teams often give you direct access to the senior people doing the work. For a first workflow, direct access and clear deliverables usually matter most.
Is a free proof of concept a good idea?
It can be, but check what it proves. A free demo on clean sample data says little about your systems. A small paid discovery with specific deliverables is often better value.
What should the contract say about intellectual property?
That you own the deliverables built for you, including code, documentation and knowledge assets, and that any pre-existing vendor components are licensed for your use. Have your counsel review the terms.
How do we compare quotes?
Normalize them to the same scope, deliverables and acceptance criteria. A cheaper quote without evaluation cases or handoff documentation is not cheaper.
Legacy System AI Discovery
Map one existing workflow, assess its sources and define a maintainable expert asset.
Explore the scope and deliverables →A FOCUSED FIRST ENGAGEMENTClinical Evidence Engineering
Design one evidence inspection module with source links and explicit review states.
Explore the scope and deliverables →