Test it on your own inbox, not on a demo
Bring real requests, and pick the messy ones: a product description instead of a SKU, a competitor’s code, a PDF from a customer’s purchasing system, a spreadsheet, a line that ends in “or equivalent”.
A tool that only shines on clean SKUs automates the part your team already does quickly. The value is in the lines someone currently has to look up.
Ask whether it handles quotes, orders, or both
A quote request asks for a price; a purchase order places one. They need different work, and a tool built for one leaves the other in the inbox.
Check the mix in your own mailboxes before you decide what to automate. In six weeks of one lighting distributor’s inbox we counted 650 purchase orders and 192 quote requests — orders outnumbered quotes about 3.4 to 1. A project scoped as “quoting” would have missed most of the volume.
Test the matching on your hardest lines
Give it near-identical products that differ by one attribute — 3000K against 4000K, IP54 against IP65. Give it the same spec written several ways, with a decimal comma, a degree sign or different units. Give it a competitor’s code and see whether it works out what that product is before choosing yours.
Include lines where the right answer is “I am not sure”. Those tell you more than the ones it gets right.
Find out what it does when it is unsure
This is the most important question. A tool that always returns a confident match will sometimes return a confident wrong one, and a wrong product on a quote is a wrong price on a document your customer may accept.
Ask how a line is marked when the tool cannot be certain, and whether an unrecognised item can ever be marked ready without a person seeing it. The answer should be a rule in the product, not a setting someone can change.
Check where the price comes from
The price on a draft should come from your ERP: this customer’s price list, their discount, your margin rules. It should never be generated by a language model.
Ask to see the arithmetic on a line — list price, customer discount, net price. If the tool cannot show it, your team cannot check it.
Decide what it may write to your ERP, and when
Reading your catalog and writing quotes into your ERP are different levels of trust. A sensible rollout reads first, drafts for review, and writes back only once the team trusts the drafts.
If the tool can send or post anything on its own, ask which request types that applies to, who chooses them, and what happens to a line that is not a clean match.
Ask the security questions before the demo, not after
Where is the data stored? Which AI models read the emails, and may they train on them? Is anything learned from your data used for another customer? Who at the vendor can see your mailbox, and is every action logged where your own admin can check it?
A vendor who answers those on a public page is easier to evaluate than one who answers them after you have signed.
Run the pilot on live requests, and measure it yourself
Start with one mailbox and real traffic, not a curated sample. Count it yourself: how many lines came out ready, how many your team had to correct, how many were flagged for a person, and how long the first reply took compared with before.
Agree those measures before the pilot starts, so the decision afterwards is about your numbers rather than the vendor’s.
The mistakes to avoid
Judging the tool on a demo with clean SKUs. Scoping the project as “quoting” when most of the inbox is orders. Connecting every mailbox and channel at once. Switching off review before the team trusts the drafts. Letting a language model set prices. Treating “it never says it is unsure” as a strength.