⚡The short version: all twelve questions
Most agency vetting fails because buyers ask about process awards, years in business, and portfolio size — things that are easy to claim and hard to verify. The questions below are different. Each one forces a concrete answer, and each concrete answer can be checked against references, contracts, or a small paid pilot before you commit real budget.
Use the table as your interview spine. The sections after it explain why each cluster matters and what to do with the answers.
| # | Ask this | A good answer sounds like | A bad answer sounds like |
|---|---|---|---|
| 1 | Who will be on my team, by name and role? | Named people, roles, allocation percentages | "A senior team will be assigned" |
| 2 | Will any work be subcontracted? | A clear yes or no, with who and where | Evasive talk about "partner networks" |
| 3 | What is the seniority mix on my account? | Ratio of senior to mid, with tenure at the firm | "We only hire seniors" — check references |
| 4 | Who writes the first lines of code, and who reviews it? | A named lead and a review process | "Our process handles that" |
| 5 | Show me a real sprint board from a comparable project | A walkthrough of an anonymized live board | A slideware diagram of "our methodology" |
| 6 | What happens when a milestone slips? | Escalation path, named owner, past example | "That does not happen here" |
| 7 | Who tells me bad news, and how fast? | PM or lead, within a defined window | "Your account manager handles communication" |
| 8 | What exactly do I own at the end? | Full IP assignment, code, docs, credentials | Licenses, escrow clauses, "our framework" |
| 9 | How do change requests work, and what do they cost? | A written change process with example pricing | "We are flexible" — flexibility is billed later |
| 10 | What happens if I leave after month two? | You keep code, docs, and a handover period | Termination fees, hostage infrastructure |
| 11 | Describe your security practices in specifics | Access control, secrets handling, device policy, incident process | "We take security very seriously" |
| 12 | Give me three references from projects like mine | Three names with permission to call | "Our clients are under NDA" — all of them, always |
The single most predictive signal in agency vetting is specificity. Every answer above has a specific version and a vague version, and the vague version is the tell — regardless of how polished the deck is.
🎯Why most agency vetting fails
Buyers typically evaluate agencies on three things: the proposal document, the sales calls, and the price. All three are produced by the sales function, not the delivery function. The people you meet before signing are rarely the people who build your product, and the proposal is written to win the deal, not to describe the engagement accurately.
The result is a selection process optimized for presentation skill. Agencies that invest in sales win more work; agencies that invest in delivery keep more clients. You want the second kind, and you cannot find them by reading proposals — you find them by asking questions that only a real delivery organization can answer.
There is also a structural problem: most buyers have never run a software project before, so they do not know what a healthy engagement looks like from the inside. The twelve questions are designed to work even if you have no technical background, because they test organizational behavior, not technical claims.
👥Questions 1–4: who actually does the work
The most common failure mode in outsourced development is the bait and switch: senior people in the sales process, junior people on the project. Question one exists to prevent it. Ask for the team by name, role, and allocation percentage before you sign, and write the named lead into the contract with a replacement-approval clause. An agency that will not name its team is telling you the team is not assigned yet — which means it is whoever is unassigned when you sign.
Question two — subcontracting — matters more than most buyers realize. Some agencies are delivery organizations; others are brokerage layers that resell subcontractor capacity with a margin. Subcontracting is not automatically bad, but undisclosed subcontracting is, because it breaks every accountability mechanism in your contract. Ask directly, and ask where the subcontractors sit.
Questions three and four probe the actual engineering practice. A real seniority mix is something like one senior lead, two mid-level engineers, and a shared QA function — not a uniform wall of "seniors". And the code review answer should name a mechanism: pull request review by someone other than the author, on every change, with the lead reviewing architecture-affecting work.
| Team signal | Healthy | Warning sign |
|---|---|---|
| Team naming | Named individuals before signature | Roles only, names "after onboarding" |
| Lead allocation | Lead at 50%+ allocation on your project | Lead spread across five accounts |
| Subcontracting | Disclosed, in the contract, with locations | Discovered after the first invoice |
| Seniority mix | Stated ratio, verifiable on calls | "All senior" at a junior price point |
| Code review | PR review by non-author, enforced | "The team reviews informally" |
📦Questions 5–7: delivery, escalation, and bad news
Question five — show me a real sprint board — is the closest thing to a work-sample test you can run on an agency. A delivery organization can open an anonymized board from a live project and walk you through how work flows, how estimates compare to actuals, and how blocked items get handled. A sales organization can only show you a diagram.
Questions six and seven are about failure, because every software project has some. The question is never whether a milestone slips — it is what the organization does next. A good answer names the escalation path: the project manager flags it in the weekly update, the delivery lead owns the recovery plan, and you hear about it within days, not at the end of the month when the invoice arrives.
The bad-news question is the one agencies least expect, and the answers are revealing. Healthy agencies have a norm that problems travel fast — the same mechanism that carries status carries risk. Unhealthy ones have account managers whose job is to keep the client calm, which means you learn about problems when they are expensive.
Ask the agency to describe the last project that went badly and what they did. An agency with no failure stories has either never delivered anything hard or will not tell you the truth — and neither answer is acceptable.
📜Questions 8–10: IP, pricing, and the exit
IP terms are where vetting turns into contract review, and they deserve more attention than they usually get. The standard you want is simple: on full payment, you own the code, the documentation, the designs, the infrastructure configuration, and every credential. Watch for two carve-outs. The first is "our proprietary framework" — pre-existing libraries the agency uses across projects, which should be licensed to you perpetually, not left ambiguous. The second is code escrow, which is often presented as protection but usually means you do not get the repository until a dispute triggers a release.
Question nine — change requests — is where budget overruns are born. Every project changes mid-flight. The agency should have a written change process: a description, an estimate, your approval, then the work. "We are flexible" without a process means the flexibility is invoiced at the end, and the end is when you have the least leverage.
Question ten — the exit — tells you whether the agency earns retention or enforces it. You want termination for convenience with reasonable notice, a defined handover period, and no termination fees beyond work performed. An agency confident in its delivery does not need to lock you in.
| Contract clause | What you want | What to reject |
|---|---|---|
| IP assignment | Full assignment on payment, including docs and credentials | License-only, escrow, retained "framework" ambiguity |
| Pre-existing code | Perpetual license to agency libraries used in your build | No mention — it becomes leverage later |
| Change requests | Written estimate and approval before work starts | "Flexible" scope with no process |
| Termination | For convenience, 2–4 weeks notice, handover included | Termination fees, 90-day notice, hostage hosting |
| Named team | Lead named, replacement requires your approval | "Resources may vary at our discretion" |
🔒Questions 11–12: security practices and references
Security answers should be boring and specific. You want to hear: named access control with individual accounts and MFA, secrets in a manager rather than in code or chat, managed devices with disk encryption, least-privilege access to production, and an incident response process with a named owner. If the agency handles regulated data — health, financial, education records — ask how that data is isolated and who can touch it, and treat "we comply with everything" as a non-answer. For regulated work, involve your own counsel; this article is not legal advice.
Question twelve — references — is the one that verifies everything else. Three references from projects comparable to yours in scope and domain is a reasonable ask. "All our clients are under NDA" is a red flag, not a constraint; NDAs cover the work product, not the existence of the relationship, and every real agency has clients who will take a call.
When you get the references, do not ask "were you happy". Use the script in the next section — it is built to surface the things references politely omit.
☎️The reference-check script
A reference call should take fifteen minutes and follow a fixed script so you can compare answers across agencies. Ask every reference the same questions, in this order, and write down the answers verbatim — the phrasing matters as much as the content.
Who was actually on your team?
Compare the names to what the agency told you. If the "senior lead" was on the account for two hours a week, you will hear it here.
What was the first thing that went wrong, and what did they do?
Every project has an answer. A reference who says nothing went wrong is protecting the relationship, not informing you.
How did the final cost compare to the proposal?
Ask for the pattern, not the number: within ten percent, within a third, or double. Then ask what drove the gap.
How did they handle a change you requested mid-project?
This tests the change-request process in practice — was there an estimate and an approval, or a surprise line item?
Did you get full control of everything at the end?
Code, documentation, credentials, infrastructure. Any hesitation here goes straight to your contract negotiation.
Would you hire them again for the same project?
The closing question. A pause before "yes" is an answer. "Yes, but for a different kind of project" is a better one — ask why.
Call at least one reference the agency did not offer — a client you find through your own network or public case studies. Curated references are marketing; uncurated ones are diligence.
🚩Proposal red flags: a preview
After the interviews come proposals, and proposals have their own tells. These are the patterns that correlate with trouble, based on what we see when clients bring us proposals to second-opinion.
Get a proposal you can compare line by lineTalk to us about a second opinion on a proposal
A price more than 40 percent below the other bids
Software development has a real floor cost. A dramatic outlier is either under-scoped, junior-staffed, or planning to make it up on change requests — all three end the same way.
A fixed price for an undefined scope
A fixed number attached to a vague scope means the vendor has priced the argument about scope, not the work. Fixed price works for precise specifications and nothing else.
No named team in the proposal
If the proposal cannot name the people, the people do not exist yet. You are buying capacity the agency hopes to have when you sign.
Discovery billed as optional
For anything beyond a small, well-defined build, skipping discovery does not save money — it moves the cost of unknowns from a planned phase into mid-project renegotiation.
Timelines with no dependency on your inputs
A timeline that assumes instant answers, perfect content, and no decision latency from your side is a timeline designed to be missed with the blame pre-assigned to you.
Case studies you cannot verify
Logos without contactable humans, metrics without a source, "leading fintech company" as a client name. Ask for the person behind each case study.
🗓️Running the process in two weeks
A disciplined vetting process does not need to take months. Week one: send the twelve questions in writing to three to five shortlisted agencies, and score the written answers for specificity before any call. The written round alone usually cuts the list in half, because vague answers are obvious on paper in a way they are not on a polished call.
Week two: video calls with the survivors — including the proposed delivery lead, not just sales — then reference checks using the script, then contract review of IP, change, and termination terms before you negotiate price. Price negotiated before terms are clear is price negotiated blind.
If the project is large enough to matter, add a paid discovery or a small pilot phase — two to four weeks of real work with real deliverables. A pilot is the only vetting step that measures the thing you actually care about: what it is like to work with this team. An agency that performs well in interviews and badly in a pilot has just saved you a year.