Summary. Boards are being handed AI return figures that nobody had to verify. We build the systems those figures are about, and the returns we can evidence are not the ones that get reported. This is an account of what is measurable, what is merely reportable, and the decision instrument we use to tell them apart.
#The question that has no owner
The question a board asks is "what ROI are we getting from AI?", as though the number existed somewhere in the building and someone had simply not written it down yet.
Nobody has it. We are usually the ones asked, and what we can hand over honestly is narrower and stranger than the question expects: a stage that used to breach its timeout and now does not, a bill that fell while volume rose, a provider that could be swapped in an afternoon. None of those are what the question was reaching for.
PwC's AI performance study puts real numbers against the gap. It surveyed 1,217 senior executives, director level and above, 91% at publicly listed companies and 76% at companies with a billion dollars or more in revenue, across 25 sectors, with 85 respondents in Africa. The top 20% capture 74% of AI-driven returns. The most AI-fit companies show 7.2 times the AI-driven performance of everyone else. Africa sits at the global median on the fitness index while trailing the leaders on every dimension of performance, which the report reads, reasonably, as an execution problem rather than an adoption one.
The survey is competently run and the recommendations are sound. The difficulty is in the methodology note, where AI-driven performance is defined as the revenue and cost gains executives attribute to AI. Sixty management practices were then tested against that figure. The outcome and the practices come from the same person in the same sitting.
In the programme we built, a participant cannot claim an outcome. An outside party confirms it, or it does not count. Almost every published AI ROI figure is reported on precisely the terms that rule exists to reject.
#What is actually being measured
| What gets reported | What was measured | The reflex | What would have to be true |
|---|---|---|---|
| "AI increased our revenue" | An executive's attribution, after the fact | Benchmark against the leaders | A pre-period, a comparable untreated case, and someone outside the team confirming the revenue |
| "AI-driven cost savings" | The distance between two estimates | Put it in the ROI model | A bill that fell on unchanged or rising volume |
| An AI fitness score | Self-described practice, regressed on self-described gain | Raise the score | An outcome measured by anyone other than the person describing the practice |
| "We have scaled AI enterprise-wide" | A count of deployments | Fund more pilots | One use case somebody outside the team would notice if it were switched off for a week |
Read the last column downward and the shape is clear. Every AI number that survives contact with an auditor has the same three parts: a baseline captured before the change, a population that did not get the change, and a party with no stake in the answer. Confidence in AI correlates with how well a company is run, and well-run companies report more of everything. A survey cannot separate the two, which is not a flaw in the survey. It is a limit on what a survey is.
#The return that looks like a retreat
The cheapest inference is the call you never make. In our multi-tenant publishing platform, candidate topics are gathered from public signals and scored deterministically before any model is invoked. The expensive stage only ever sees material already judged worth paying for.
That single decision moves more money than any model upgrade we have made. It also arrives on the board's desk as a falling AI bill, which reads as reduced AI investment, which reads as a lower fitness score.
Here is the cost asymmetry. A model upgrade that improves output by a few per cent is legible upward: it has a vendor, a launch date and a name. An architectural filter that removes most of the calls has none of those, and its evidence is a line going down. The framework rewards the first and is structurally blind to the second, so the organisations best at controlling AI cost are the ones most likely to be told they are underinvesting.
#Latency is a budget, and a budget is the only auditable AI number you own
A ceiling turns an improvement into a fact. Extraction in our financial intelligence platform ran per article, sequentially, and exceeded the pipeline's timeout. Restructuring the batch to run concurrently cut extraction time by roughly two thirds and brought the stage back inside budget.
Compare the evidentiary status of that with "AI helped us grow revenue". The extraction figure has a fixed corpus, a before, an after, a threshold that was breached and then was not, and a re-run anyone can repeat. It is falsifiable. Somebody can take the same inputs and show we were wrong.
The relocation worth making is this: performance work gets filed under engineering, and revenue attribution gets filed under strategy, when the evidentiary quality runs exactly the other way. The engineering number is the one that would survive a hostile reading. The strategy number would not survive a curious one.
This is why we set a latency ceiling before choosing an architecture rather than after. A ceiling is not only a service commitment. It is the instrument that converts a change into a measurable claim, because a threshold gives you a pass and a fail where a percentage only gives you an argument.
#The price of a model is what it costs to leave
Schema is the contract. When we moved entity extraction and sentiment off locally hosted transformer models and onto a hosted LLM, the interfaces and data structures stayed identical, and every downstream consumer kept working through the migration. Multiple providers now sit behind one interface, selected by configuration. A provider that degrades or reprices is a config change; a provider that disappears degrades the feature instead of taking the service down.
# the paid stage is the only stage that knows a provider exists,
# and it is not allowed to change the shape of what it returns
candidates = gather(sources) # free
scored = [c for c in candidates if worth_it(c)] # deterministic, free
results = extract(scored, provider=CONFIGURED) # metered
assert shape_of(results) == EXTRACTION_CONTRACT # true for every provider
What the shape buys is two things, and the second is the point. It makes the cost of the metered stage a function of a threshold we control rather than a vendor's price list. And it makes the exit cost knowable, because nothing downstream knows which path produced the data.
Now add the term the study does not carry. Metered inference is denominated in dollars. Revenue across the markets we build for is not. A use case that clears its hurdle rate this quarter can go underwater on a devaluation with no change in usage, no change in performance and no decision made by anyone in the company. Priced properly, an AI use case in this region is a position with currency exposure, not a project with a payback period, and "self-host where the volume justifies it" is a hedge before it is a cost decision.
#What we got wrong
For months we treated a data-modelling problem as a performance problem. The core object in our financial intelligence platform is one company supplying another, which is a graph, and we had modelled it in whatever store happened to be nearest. Every question worth asking became a recursive query that got slower exactly as the data got more valuable. We tuned it. We changed environments. The bottleneck followed us across all of them, and the fix in the end was a data model rather than a platform. That migration is documented in full in What the Free Tier Actually Costs.
The part worth extracting here is why it survived that long. Each change produced a fresh set of numbers with nothing to hold them against: no frozen query set, no fixed corpus, no preserved before. Everything looked better in ordinary use after every change, because effort had gone into it and effort usually shows. What we never had was a comparison capable of telling us that the improvement was real but the diagnosis was wrong.
The mechanism is not carelessness. Baseline capture competes for time with the fix itself, at the exact moment the team is most certain the fix will work, and certainty is what removes the incentive to measure. That is also how organisations end up reporting AI gains nobody can verify: one justified skipped baseline at a time, until a board asks for the number and the only honest answer left is a recollection.
The remedy is procedural rather than technical. A frozen input sample is captured when a use case is approved, not when it is optimised, and it lives with the use case for as long as the use case does.
#Before you believe your own AI number
| Question | If yes | If no |
|---|---|---|
| Do you hold inputs from before the change, and can you re-run them? | You can produce a number | You have a recollection; capture the sample today |
| Would anyone outside the delivery team notice if it were switched off for a week? | It is deployed | It is a pilot with a production URL |
| Does the bill move with volume, and do you know the shape of the curve? | You can forecast | You are carrying unpriced exposure |
| At what exchange rate does this use case stop clearing its hurdle? | Price it and hedge it | The return is a currency bet nobody has authorised |
| Could you change providers with a configuration change this week? | Your exit cost is known | The vendor sets your future ROI, not you |
| Is there a written list of what the model may never decide? | You can defend it to a regulator or a partner | The limit is whatever the last engineer assumed |
The row most teams skip is the exchange rate one, and it is the row that decides whether the others matter.
#When to skip this
- You are a listed firm with a portfolio of use cases and a real capital allocation process. PwC's advice is aimed at you and lands cleanly. Portfolio discipline beats measurement rigour when you already have thirty bets running.
- You are still looking for product-market fit. Baselines, holdouts and frozen samples are overhead on a system whose shape will change next month. Ship, watch, and start measuring when the use case stops moving.
- Your inference spend is small next to payroll. The currency term is real but immaterial. Price it once, then stop thinking about it.
- The use case is genuinely new revenue rather than a substitution. There is no clean baseline to capture, because there is no before. Say that plainly rather than manufacturing a comparison.
- Our own evidence has limits. This is drawn from a handful of production systems with no control group. It is a different class of evidence from a survey of 1,217 executives, with its own standards, and it does not license a claim about what the average African company should do.
#Key takeaways
- Capture the frozen input sample when the use case is approved, not when it is optimised. The before disappears quietly and permanently.
- Set the latency ceiling before choosing the architecture. A threshold converts an improvement into a claim somebody else can check.
- Report the calls you never made as a return, in the same paper as the calls you did, or your best cost decision will read as underinvestment.
- Price every metered use case at a stressed exchange rate and state the breakeven. An unhedged AI budget in local-currency revenue is a position, not a plan.
- Keep the extraction contract identical across providers, so the exit cost stays knowable while you still have the option to use it.
- Treat any AI return figure without a baseline, a comparison and an outside confirmation as a belief. It may well be a correct belief. It is not a number.
Drawn from building and running AI systems in production across financial intelligence, multi-tenant publishing and programme delivery, in markets where inference is billed in dollars and revenue is not.