Short summary
A black-box AI system cannot tell you why it reached a conclusion. In regulated operations — where every material decision must be explainable, auditable, and attributable to an authorised person — that opacity is a disqualifying deficiency.
- What black-box AI means and why it is incompatible with regulated operations
- The explainability requirements from EU AI Act, sector regulators, and CSRD
- The audit test: what questions you must be able to answer about every AI-assisted decision
- How governed AI — acting inside rules with a full audit trail — provides the alternative
The term "black box" is used loosely in AI discussions. For regulated businesses, it has a precise meaning: an AI system whose reasoning process cannot be inspected, explained, or traced to a decision rationale that a human can review and take responsibility for.
Black-box AI is not prohibited in all contexts. A recommendation algorithm that suggests which article to read next can be opaque without consequence. But an AI system that assists in approving a supplier emission declaration, classifying a transaction for carbon accounting, or generating a climate risk assessment is operating in a context where explainability is not optional — it is a regulatory requirement.
What Black-Box AI Actually Means
A black-box AI system is one where the relationship between inputs and outputs cannot be explained in terms that a human reviewer can understand and verify. Most large language models, deep neural networks, and ensemble models are partially or fully black-box in this sense: they produce outputs with high confidence, but cannot generate a step-by-step account of how they reached them that is both accurate and comprehensible to a non-specialist.
The contrast is a governed AI system: one that operates within explicit rules, applies those rules to inputs in a traceable way, and produces an output accompanied by a record of what rules were applied, what inputs were used, and what the system concluded and why. The AI may still be sophisticated, but the governance layer makes its actions auditable.
The Regulatory Explainability Requirement
EU AI Act. High-risk AI systems (which include AI used in AML, credit decisions, recruitment, and some compliance workflows) require technical documentation that allows assessment of compliance, human oversight measures, and the ability for operators to explain individual outputs. From August 2026, deployers of high-risk AI must implement these requirements.
Sector regulators. The FCA (UK), ECB/EBA (EU), and equivalent regulators have consistently stated that algorithmic decisions in regulated contexts must be explainable to the regulator, to affected parties, and to internal governance. The FCA's guidance on model risk management requires that firms understand how their models work and can explain their outputs. "The model decided" is explicitly not sufficient.
GDPR Article 22. Automated processing that produces decisions with significant legal or similar effects on individuals requires a human in the loop — or explicit consent and the ability to explain the decision logic. Any AI-assisted decision that affects a client, employee, or counterparty may trigger Article 22.
CSRD assurance. Where AI is used to assist in calculating, classifying, or summarising sustainability data for CSRD disclosure, the assurance provider will ask how the AI-assisted output was generated, who reviewed it, and how errors would be detected. If the AI is opaque, the assurance provider cannot assess the reliability of the output.
The Audit Test: Can You Explain This Decision?
When a regulator or assurance provider asks about an AI-assisted decision, they will ask a sequence of questions:
- What inputs did the AI system use to reach this conclusion?
- What rules or model logic were applied to those inputs?
- What did the system output, and in what format?
- Who reviewed the output before it was acted upon?
- What authority did that person have to approve the action?
- Is the action recorded in the audit trail with timestamp and approver identity?
A black-box AI system can answer none of the first three questions satisfactorily. A governed AI system answers all six.
Governed AI: Acting Inside Rules
Governed AI is AI that operates within an explicit rule and permission framework. It does not act autonomously — it acts within the boundaries defined by your governance model. Its actions are logged with the inputs, the rules applied, the output generated, and the human who approved the action. The audit trail is the governance record.
This is not a technical constraint — it is an architectural choice. A governed AI system can be just as capable as an ungoverned one. The difference is that its capabilities are exercised within a permission model, and its outputs are subject to human review before they have effect.
HubAI: Governed AI Inside Your Compliance Boundary
HubSecure's HubAI operates inside your permission model. Every AI action — summarising a supplier declaration, flagging an anomaly in emissions data, generating a draft report section — is logged with the prompt, the model used, the output, and the user who triggered it. AI does not approve. People approve. AI assists within the rules.
Building Explainability Into AI Operations
- Classify every AI use case in your organisation by risk level (EU AI Act framework) and explainability requirement
- For high-risk use cases, implement a human-in-the-loop requirement: AI assists, human approves
- Log all AI actions with: input summary, model/system used, output, timestamp, triggering user
- Build a review workflow: AI output goes into a review queue; a qualified person reviews before the output is accepted
- Document the AI system for each use case: what it does, what rules it applies, what inputs it uses
- Test explainability: run a simulated audit and answer each of the six audit test questions for each AI-assisted process
Governed AI for Regulated Operations
HubAI operates inside your compliance boundary — 71 tools, 34 models, full audit trail. Every AI action is logged with the user, timestamp, input, and output. AI assists; people decide.