Black box in, black box out: an AI agent cannot explain a score it cannot read
Hand an agent a number with no reasons and it will supply its own. They will sound right, and they will not be why the record was chosen.
Everyone in data knows the old line: garbage in, garbage out. There is a version for AI agents that gets less attention. If the score you hand an agent has no reasons attached, nothing the agent does with it will have reasons either. Black box in, black box out.
This is an opinion piece, and the opinion is simple. An agent cannot explain a score it cannot read. Stacking a second thing you cannot see into on top of the first does not average out the problem. It makes the problem worse, and in one specific way that is easy to miss.
The stack, from the bottom
Picture the usual setup. A scoring tool puts a number on each account. An AI agent reads the number, decides what to do, and writes an email. A buyer gets the email.
Now ask the question a sales manager asks every week: why did this account get this email?
- The email exists because the agent chose the account.
- The agent chose the account because the score was 82.
- The score was 82 because the model said so.
The chain ends there. Two steps back from a real message to a real buyer, and there is nothing left to read.
The part that is easy to miss
You would expect a black box to give you silence. It does not. Ask a language model why an account matters and it will usually tell you, fluently, whether or not it was given a reason.
Hand an agent a bare 82 and ask it to write a relevant email, and it will find something to say. It will read the website, pick a detail, and build a case. The case may be good. It may also have nothing to do with why the account scored 82, because the agent was never told.
So the buyer gets an email with a confident reason in it, the manager reads the email and sees a reason, and everyone assumes the reason came from the score. It did not. It was made up after the fact to fit a number.
That is what stacking does. One black box gives you a number you cannot explain. Two give you a number you cannot explain and an explanation nobody can check.
Three ways it shows up
You cannot tell which part was wrong. A bad email went to a bad account. Was the score off, or did the agent misread it? With no reasons on the score, both look the same from outside.
You cannot fix it in the right place. If the problem is that company size counts for too much, the fix belongs in the model. If you cannot see the model, the fix goes into the agent’s prompt as a patch, and the next agent you build does not get it.
You cannot defend it. When a rep, a manager, or a customer asks why, “the model said so” is the whole answer. People stop trusting things that answer like that, and they are right to.
The why-chain test
Here is a test that takes about ten minutes. Pick one email an agent sent last week. Ask why, three times.
| Ask | A readable answer | A black box answer |
|---|---|---|
| Why this account? | “It scored 7.3, and we work 6.5 and up.” | “The agent picked it.” |
| Why that score? | “Company size and region are strong. Contacts in buying roles is weaker.” | “That is what the model returned.” |
| Why do those factors count? | “We set company size at 25% because our wins cluster there. It is written down.” | “We do not know what it weighs.” |
If you get to the third row with real answers, the chain is readable. If it breaks at the first row, the agent is the black box. If it breaks at the second, the score is. Most teams find it breaks at the second, and that they had been reading the agent’s emails as if they were the answer to it.
You only need to fix the first link
The good news is that you do not have to make everything readable. An agent built on a language model will always be a little hard to see into. That is fine, as long as two things are true:
- What goes in is readable. The score comes with its factors and weights, on the record.
- What it may do is written down. The rules for acting on the score are a short list a person wrote.
With those two, the agent’s job shrinks to something you can check: did it follow the rules, using the reasons it was given? You are no longer asking a model to account for its own judgment. You are checking its work against a page.
- Industry 30% 7.9
- Company size 25% 8.7
- Contacts in buying roles 25% 6.4
- Region 15% 9.1
- Existing partner 5% -3.8
The math: 7.9 at 30% is 2.37, 8.7 at 25% is 2.175, 6.4 at 25% is 1.6, and 9.1 at 15% is 1.365. That adds to 7.51. The disqualifying factor subtracts 3.8 at 5%, or 0.19, for 7.32, shown as 7.3.
An agent given this does not need to invent a reason. It has four, with their sizes.
The fair objection
A black box score can be accurate. A trained model with years of clean history may rank accounts better than any weights a person would set. If the score only sorts a queue, and no person or agent has to say why, that may be all you need.
Agents change the picture because they do more than sort. They write, and writing needs a reason. Give an agent an accurate number with nothing behind it and you get accurate targeting with invented messaging. For some teams that trade is acceptable. It should be a choice, not a surprise.
Where BoostIdeal fits
BoostIdeal is built to be the readable first link. You set the weights, with AI helping you analyze your data and draft. Every score is a weighted sum out of 10, and the factors behind it are written to the record in HubSpot, where your agents and workflows can read them. It can also share those reasons wherever your rep works, such as in Slack or by email. The weights do not change unless a person changes them.
It is not the agent, and it does not run one. It also does not keep a score history, so the chain it gives you is for the score as it stands now, with the time it was last calculated.
To write the model that sits at the bottom of the chain, start with your ideal customer profile and the five-step framework for building an ICP scoring model.
Frequently asked questions
What does “black box in, black box out” mean?
If the score going into an AI agent has no visible reasons, the agent’s decisions and messages have none either. The agent may still produce an explanation, but it is not the real reason the record was chosen.
Can an AI agent explain a lead score?
Only if the reasons are given to it. An agent handed a bare number can describe the account, but it cannot know why the number is what it is.
Is a black box score always a problem?
No. If the score only sorts a list and nobody needs the reasons, accuracy may be enough. It becomes a problem when a person or an agent has to act on why.
How do I check my own setup?
Pick one email an agent sent, and ask why three times: why this account, why that score, why those factors. Note where the answers stop.
The takeaway
An agent passes along what it is given. Give it a number with no reasons and it will supply its own, and they will sound right. Make the first link readable, write down what the agent may do with it, and the rest of the chain can be checked.