Everyone agrees the software looks good, but has anyone asked whether it solves the problem your team actually has?
The product looks finished, the presenter knows every shortcut, and the sample data fits perfectly. If your team hasn't written down what it needs before that first demo, the demo writes it for you. Every later vendor gets judged against the first one.
The fix starts before a presentation, when you name the decision owner and choose one real workflow every finalist must support. Set must-haves and weighted criteria before any vendor demo. Screen out products that fail hard constraints, test the rest on the same workflow, and record the evidence behind each score. Treat unverified claims as open questions, never proven capabilities. Decide based on observed results, total cost, and documented trade-offs.
Consider a 60-person company replacing an email-and-spreadsheet process for new-vendor setup requests. Its operations lead needs a way to compare software when evidence is incomplete, scores are close, and no vendor is perfect. The method below follows that selection from written criteria through a documented decision.
Why Software Selections Go Wrong Before the First Demo
The costliest selection mistakes happen long before the purchase, in four recognizable ways.
Criteria get written after the team has a favorite. Someone sees a polished demo, the team likes it, and the requirements list quietly takes the shape of that product. Every other vendor is then scored against a feature list written by its competitor.
Requirements are feature lists. "Has approval workflows" is a feature. "Routes a vendor setup request to the right approver based on its type, without the requester choosing" is a requirement you can test. Feature lists favor whichever product has the longest one; requirements favor whichever product handles your work.
Nobody owns the decision. When the person scoring, the person paying, and the person approving are never named, the loudest stakeholder decides, and the decision gets reopened every time someone new disagrees.
Scores have nothing behind them. A 4 out of 5 from a vendor's slide and a 4 out of 5 from your own team's pilot look identical on a spreadsheet. They mean different things, and a scorecard that doesn't record the difference can't be trusted when the totals are close.
Each stage below fixes one of these before it can shape the next.
Define the Decision Before You Look at Vendors
Before anyone contacts a vendor, write a one-page decision charter. It answers six questions:
- What problem are we solving? One or two sentences, in operational terms.
- Which workflow will every finalist be tested on? One real process, with real request types and real exceptions.
- Who decides? Name the person who owns the recommendation, who confirms the budget, and who approves the purchase. Consulting, scoring, and approving are three different roles, even when one person holds two of them.
- What's the budget range? A range, including implementation and the first year of administration.
- What are the hard constraints? Security and data requirements, regions, existing systems the tool must connect to, and anything else that eliminates a product outright.
- Who gives input, and how? Each stakeholder's role, so input arrives through the criteria and the scoring, never through a hallway veto at the end.
In the worked example, the operations lead drafts the charter before any vendor call. The problem: new-vendor setup requests arrive by email with missing information, wait in inboxes, and get chased by hand. The workflow under test: a new-vendor setup request, from submission to a vendor ready to pay. The roles: "The operations lead owns the recommendation, finance confirms the budget, and the designated approver authorizes the purchase." Procurement and finance give input through the criteria; neither can veto a finalist after scoring without pointing to a criterion.
Settling these roles early gives the team a reference point whenever a new stakeholder joins the conversation.

Turn Requirements into Weighted Criteria
Criteria come in two kinds, and they're scored differently.
Must-haves are gates. A finalist either meets each one or leaves the selection. Must-haves never get weights, because a high score elsewhere can't make up for a missing gate. Keep the list short; every must-have is a reason to eliminate a product, so each one has to be something the workflow can't run without.
Weighted criteria separate the finalists that pass. Each gets a weight, and the weights total 100%. Set the weights before any demo, with the stakeholders who gave input, and record them in the charter. Changing a weight after the demos is allowed only with a written reason.
Write every criterion as behavior you can observe. If what operations software should handle for a growing company is still an open question for your team, settle that first; the criteria come from it.
The worked example's criteria, agreed before any vendor call:
|
Criterion |
Type |
Weight |
|
Required fields enforced at submission |
Must-have |
Gate |
|
Routing by request type, without the requester choosing |
Must-have |
Gate |
|
An audit trail of every status change |
Must-have |
Gate |
|
Routing rules a non-developer can change |
Weighted |
25% |
|
Administration effort for the operations team |
Weighted |
20% |
|
Visibility of open and returned requests |
Weighted |
20% |
|
Field-level permissions for finance data |
Weighted |
15% |
|
First-year total cost |
Weighted |
20% |
|
Total |
100% |
Build the Long List and Cut to Three
A long list of six to ten products is enough. Build it from a few sources: a category overview such as Best Workflow Platforms guide, peers at similar companies, or the tools your existing systems already integrate with. Review sites help you find names; they can't tell you how a product handles your workflow.
Then cut on hard constraints only. Each product either meets the charter's constraints or drops off:
- Pricing model: per user, per app, platform minimums, and whether the budget range survives the first year.
- Data limits: record volumes, storage, and anything that caps the workflow at your scale.
- Security and region: where data is stored, and the certifications or agreements your organization requires.
- Existing systems: whether the product connects to the systems the workflow touches, at a plan you'd buy.
Don't score anything yet. The cut uses published facts and short vendor answers; scoring waits for testing.
In the worked example, the long list has eight products. Pricing model and data limits remove five, leaving three finalists, which the rest of this guide calls A, B, and C. All three appear able to handle vendor setup requests. "Appear" is the right word at this stage: nothing has been tested yet.
Test Every Finalist on the Same Workflow
Every finalist runs the same script on the same workflow, and the team runs at least one pilot. That's what makes the scores comparable.
The scripted demo. Send each vendor the workflow, the request types, and the must-haves before the demo, and ask to see your process run on your own example requests. The script includes at least one exception, such as an incomplete request coming back for missing information, because exceptions are where products differ.
The pilot. A demo shows what the vendor can do with your process. A pilot shows what your team can do with it. Give the strongest one or two finalists a short pilot run by the people who'll own the tool, on real requests.
For the specific tests and how to run them, see the tests and pilot design that show whether a workflow platform fits your operations team. This guide picks up where those tests produce results: turning what you observed into evidence you can score.
In the worked example, the script runs a new-vendor setup request end to end, including one request returned for a missing tax form. All three finalists run the same script. Finalist A fails a must-have: routing depends on the requester choosing the queue. Finalist B passes the routing gate, since requests route by type automatically, but a vendor engineer makes the routing-rule changes in the demo, so whether the team's own staff could make them stays unproven. Finalist C handles the full script, so the team gives C the buyer-operated pilot and runs it for two weeks on live requests. Piloting one finalist is a legitimate choice, as long as the team records what the others haven't been tested on. B stays in the selection with that gap on record.
Score What You Observed
A score is only as strong as the evidence behind it, and no single kind of evidence is strongest for everything. Each source proves some things and leaves others open.
|
Evidence source |
Strongest use |
What it doesn't prove |
|
Vendor documentation |
Published functionality, stated limits, commercial terms |
How it performs in your workflow |
|
Generic demo |
A first understanding of the product |
Fit for your process |
|
Scripted demo on your workflow |
How the product handles your process |
Whether your team can maintain it |
|
Buyer-operated trial or pilot |
Usability, configuration effort, and behavior under tested conditions |
Long-term reliability or future commercial terms |
|
Comparable customer reference |
Life after launch: support, adoption, operational burden |
Current functionality or identical results |
The rule that makes a scorecard trustworthy: every score states what was tested, what the evidence supports, and what remains unproven.

Score weighted criteria on a defined scale. This rubric is a suggested starting point; adapt the wording to your criteria:
|
Score |
Meaning |
|
1 |
Can't meet the criterion in the tested configuration |
|
2 |
Meets it only with substantial workarounds or unacceptable effort |
|
3 |
Meets the basic requirement with manageable limitations |
|
4 |
Meets the requirement with minor limitations |
|
5 |
Fully meets the defined requirement under tested conditions |
Six scoring rules keep the result reproducible:
- Must-haves are pass/fail gates, kept separate from weighted criteria.
- Weighted criteria total 100%.
- Every scored criterion uses the defined scale and cites observable evidence.
- An untested criterion is marked unproven. Unproven is neither a zero nor a pass; it holds the evaluation open until the criterion is tested, waived by an explicit decision, or recorded as an accepted risk.
- Reviewers score independently first, then record the reasons for any disagreement.
- The final choice weighs the totals together with unresolved risks and trade-offs. A small numerical gap never decides it alone.
Here is the worked example's complete scorecard. Copy the structure; one column group per finalist.
|
Criterion |
Gate or weight |
Finalist B |
Finalist C |
Evidence (B / C) |
What remains unproven |
|
Required fields enforced at submission |
Must-have |
Pass |
Pass |
Scripted demo / pilot |
B's behavior at higher volume |
|
Routing by request type |
Must-have |
Pass |
Pass |
Scripted demo / pilot |
None for the gate |
|
Audit trail of every status change |
Must-have |
Pass |
Pass |
Scripted demo / pilot |
None for the gate |
|
Routing rules a non-developer can change |
25% |
Unproven |
4 |
Vendor engineer in demo / pilot |
Whether B's staff could make the change |
|
Administration effort |
20% |
3 |
4 |
Scripted demo / pilot |
B's weekly upkeep in practice |
|
Visibility of open and returned requests |
20% |
4 |
4 |
Scripted demo / pilot |
None material |
|
Field-level permissions for finance data |
15% |
4 |
3 |
Scripted demo / pilot |
None material |
|
First-year total cost |
20% |
5 |
3 |
Written quotes |
Cost of outside help for B's routing changes, if needed |
|
Weighted total |
100% |
3.00 weighted points from 75% of criteria; full total ranges from 3.25 to 4.25 |
3.65 |
Finalist A doesn't appear: it failed a must-have and left the selection at the gate. B's total can't be completed until its unproven criterion is tested, waived, or accepted as a risk, so the scorecard shows the range it could land in.
When Scores Are Close or Evidence Is Missing
This is the part of a selection where the method earns its keep. Two finalists end up a few points apart, or the strongest one still carries an unproven criterion. The scorecard has done its job. Now the evidence behind the numbers has to decide.
When two vendors score about the same, inspect the evidence behind each score, resolve or accept every unproven criterion, weigh material differences in cost and implementation risk, and record the trade-off. Never declare a winner from a small numerical gap.
Work through it in four steps:
- Read the evidence column before the total. A 4 backed by your own pilot is stronger evidence than a 4 from a generic demo. Two equal totals built on different evidence aren't equal.
- Deal with every unproven criterion. Test it, waive it with a written reason, or record it as an accepted risk with a named owner. An unproven criterion that carries real weight can't sit in the scorecard as a blank.
- Weigh cost against implementation risk. A lower quote that depends on outside help for routine changes carries a cost the quote doesn't show. Put that cost in writing before comparing.
- Pause when a gap is material. If the missing evidence could change the outcome, run the test whenever its cost is small next to the cost of choosing wrong. If it couldn't, record the gap and decide.
In the worked example, the scorecard shows exactly where the decision turns. C's total is 3.65. B earns 3.00 weighted points from the 75% of weight it was tested on, and its full total lands between 3.25 and 4.25 depending on the routing criterion nobody has tested. If B scores 3 or higher there, B leads; at 2 or lower, C leads. The unproven criterion decides the outcome, which makes it material.

The decision owner has two defensible paths. One: recommend C on its demonstrated fit, recording B's untested routing capability as unresolved evidence and C's higher cost as an accepted trade-off. Two: give B the same routing-rule test the pilot gave C, then score both on equal evidence. C's recommendation rests on what the team has demonstrated, and the record should say so. The path to avoid is treating B's partial total as final, with a 25% criterion still blank.
Published head-to-head comparisons show the same discipline applied to two specific platforms, such as how Kintone and Airtable compare on the same operations workflow.
Make the Decision and Write It Down
Check references that match your situation. A reference is strongest on life after launch, so ask what the demo can't show: How long did setup take, and who did it? What does your team maintain each week? What did support handle well, and what did it miss? What would you configure differently? A reference from a company with a different workflow or scale proves little about yours.
Write the decision record. Five lines, filed where the next person will find them:
|
Record |
Worked example |
|
What we chose, and the date |
Finalist C, recommended to the approver on the decision date |
|
Why: the two or three criteria that decided it, with their evidence |
Routing changes by a non-developer and lower administration effort, both observed in the pilot |
|
What we gave up, and who agreed to that trade-off |
A higher first-year cost than B; finance confirmed it fits the budget range. B's routing capability remained untested when the recommendation was made |
|
Open risks, and who owns each |
Behavior at higher request volume, owned by the operations lead; retest at three months |
|
What would make us reopen the decision |
Volume doubling, a new request type the routing can't handle, or a price change at renewal |
The record does two jobs. It lets an approver sign off on reasoning instead of a number, and it lets the next person understand why the choice was made long after the team that made it has moved on.
When no finalist passes. Sometimes every finalist fails a must-have, or the gaps are too large to accept. That's a valid result. Revisit the must-haves first: if one turned out to be a preference, say so in writing and rerun the cut. If the must-haves hold, widen the long list or wait. Buying the least-bad option because the project is due creates the problem the selection was meant to solve.
When a consultant earns the fee. This method is built for a team that can run its own selection. Outside help is worth paying for when many stakeholders with competing requirements need a neutral facilitator, when regulated data adds compliance review the team can't staff, when the contract is large enough to need negotiation support, or when nobody inside has the time to own the process for its full length.
Put Kintone through the same method. Kintone is one of the platforms you can run this process on: bring your charter, criteria, and test workflow, and score it on what you observe. Book a demo with a Kintone expert and bring your scorecard.
About the Author



