Independent Validation: Why Compliance Is Not Enough
Compliance tells an organisation what it must do.
Certification confirms that defined requirements were assessed within a stated scope.
Independent validation answers a harder question:
Can the claim survive scrutiny from someone who did not design the product, operate the control, build the model or benefit from the answer?
That distinction matters. Boards, regulators, investors and procurement teams do not make consequential decisions based solely on intention. They need evidence that a product works, a control operates, a model remains reliable and a supplier’s claims are supportable.
Independent validation turns assertion into decision-grade evidence.
Claims are not evidence
Technology providers routinely describe products as secure, compliant, certified, validated, resilient or ready for regulated markets.
Those terms are not interchangeable.
A supplier may genuinely believe that its product meets a standard. That remains a first-party assertion until the relevant evidence has been examined against defined criteria.
NIST makes this distinction explicit for FIPS 140-3. A vendor may believe that a cryptographic product complies with the standard, but the module is not validated until it has undergone independent testing by an accredited laboratory and review through the Cryptographic Module Validation Program. A validated module receives an official certificate number.nist+1
The same structural distinction applies beyond cryptography:
A policy is not evidence that a control operates.
A completed checklist is not an independent assessment.
An algorithm certificate is not a FIPS 140-3 module validation.
A SOC 2 report is not a product security certificate.
ISO certification does not prove that each product produced by the organisation is secure.
Regulatory compliance does not establish fitness for purpose.
A penetration test does not validate the entire development lifecycle.
A supplier declaration does not constitute third-party assurance.
The issue is not whether the organisation is acting honestly. The issue is whether the claim has been subjected to competent challenge by a party without responsibility for producing the outcome.
Compliance, certification and validation
These terms perform different functions.
Regulation
Regulation creates legally enforceable obligations. It defines what organisations must do, prohibit or demonstrate within a jurisdiction or regulated activity.
Regulation is the rule. It is not the evidence that the rule has been followed.
Compliance
Compliance is the condition of meeting applicable legal, regulatory, contractual or internal requirements.
An organisation may assess its own compliance, commission an external review or undergo regulatory examination. The word alone says nothing about the assessor’s independence, the depth of testing or the quality of the evidence.
Certification
Certification is written assurance from an independent body that a defined product, process, service or management system meets specified requirements. ISO does not certify organisations. Certification is performed by external certification bodies, while accreditation provides independent confirmation of a certification body’s competence.iso+1
A certificate is bounded by its scope. It does not automatically cover other entities, sites, products, services, jurisdictions or technical claims.
Independent validation
Independent validation tests whether a product, model, control, system or claim is suitable for its intended use and supported by adequate evidence.
It should examine assumptions, boundaries, implementation, test methods, dependencies, exceptions and conditions of use. It should also identify what remains unproven.
Validation does not replace certification or regulatory approval. It strengthens the evidence presented to those decision processes.
Assurance
Assurance is the broader process of increasing confidence in information, controls or claims through structured evidence gathering and evaluation.
Reasonable assurance is a high level of assurance, but it is not certainty. Sampling, judgement, materiality and inherent limitations remain.
Why independence changes the answer
Product teams, model developers and control owners have valuable technical knowledge. They also carry delivery deadlines, commercial commitments, prior design decisions and reputational exposure.
Those pressures create self-review and confirmation risks even where conduct is entirely honest.
Independence changes the structure of the decision.
An independent validator should be able to:
Select evidence without relying solely on management-curated material.
Challenge assumptions and test contrary explanations.
Examine raw data, systems, configurations and records.
Report adverse findings without commercial pressure to approve.
Define limitations and conditions of use.
Escalate unresolved issues to the appropriate authority.
Remain separate from downstream implementation revenue.
External status alone does not establish independence. A consultancy that designed the product and expects to sell remediation work may face a stronger conflict than a properly separated internal validation function.
Independence must exist technically, managerially, financially and in the eyes of an informed user.
Banking already understands this
Banking regulators do not permit developers of material capital models to remain the sole judges of whether those models are reliable.
The Basel Framework requires internal models to be validated by suitably qualified parties independent of the model-development process. The objective is to determine whether each model is conceptually sound and captures material risks.bis+1
The UK Prudential Regulation Authority requires banks using internal models for regulatory capital to maintain ongoing, independent and effective challenge. Its expectations extend to vendor models because buying a model does not transfer accountability for understanding its assumptions, limitations and performance.bankofengland.co+1
This reflects a basic governance principle.
A model can be mathematically correct but operationally unsafe because:
Its data are incomplete or unrepresentative.
Its assumptions no longer hold.
Its implementation differs from the approved methodology.
Users apply it outside its intended conditions.
Portfolio or market characteristics have changed.
Overrides conceal deteriorating performance.
Supplier opacity prevents effective challenge.
Independent validation examines the relationship between the model, its data, its implementation, its users and the decision it supports.
That principle applies equally to cybersecurity products, AI systems, cryptographic implementations, cloud services and operational resilience programmes.
Failure is rarely just technical
Knight Capital lost more than $460 million after a faulty software deployment generated millions of erroneous market orders. The US Securities and Exchange Commission identified inadequate technology governance, deployment controls, supervisory procedures and effectiveness reviews. The subsequent settlement included a $12 million penalty and an independent review requirement.sec+1
TSB was fined £48.65 million by UK regulators following operational risk and governance failures connected with its 2018 technology migration. The PRA later found that TSB had not obtained sufficient assurance from its critical supplier before readiness assurances were given to the board.bankofengland.co+1
These were not simply software defects.
They involved weak challenge, incomplete evidence, supplier dependence, governance failure and decisions made without sufficient assurance.
The lesson is structural: when the consequences are material, the people responsible for delivery should not be the only people determining readiness.
FIPS, SOC 2 and ISO
Independent validation does not produce the same output in each framework.
FIPS 140-3
FIPS 140-3 applies to cryptographic modules. Accredited Cryptographic and Security Testing Laboratories test modules against defined security requirements, after which CMVP reviews the submission and issues the formal validation.nvlpubs.nist+1
A product using approved algorithms is not automatically FIPS validated. A product incorporating a validated module is not necessarily validated as a complete product.
The certificate, module version, operational environment and cryptographic boundary must match the implementation being purchased.
SOC 2
SOC 2 is an independent attestation examination of a service organisation’s system description and controls against selected Trust Services Criteria.
Its value depends on the system boundary, examination period, selected criteria, subservice organisations, customer responsibilities, exceptions and tests performed.
A SOC 2 report can provide valuable evidence. It does not prove that all services, configurations or product claims are secure.
ISO certification
ISO certification confirms that a defined management system, product, process or service meets specified requirements within the certification scope.
Accreditation assesses whether the certification body is competent and impartial for specified activities. It does not certify the certification body’s clients.ukas+1
An ISO/IEC 27001 certificate concerns the information security management system within scope. It does not establish that each product is secure or that an incident cannot occur.
Why validation should come first
Independent validation should not precede requirements, engineering, internal testing or management responsibility.
It should precede formal reliance.
That means before:
A product is presented to regulated customers.
A certification body begins its formal assessment.
A regulatory submission is made.
A board authorises launch or migration.
A customer contract contains security or compliance warranties.
An investor relies on technical claims.
A procurement team approves operational dependence.
A model influences capital, pricing or risk decisions.
This sequence matters because certification and authorisation are decision gates. They should not become expensive discovery exercises.
Independent pre-validation can identify:
Incorrectly defined scope.
Missing or unsuitable criteria.
Unsupported claims.
Incomplete evidence.
Product and system boundary errors.
Weak test coverage.
Uncontrolled dependencies.
Conflicts between documentation and implementation.
Findings likely to block certification or procurement.
Changes that would invalidate earlier results.
Early challenge is cheaper than late rejection.
NIST’s economic study of inadequate software testing estimated an annual US cost of $59.5 billion in its 2002 economic base, with $22.2 billion in feasible improvement benefits. The figures are historical, but the underlying mechanism remains relevant: defects become more expensive when discovered after release, integration or customer dependence.
What good validation looks like
A credible independent validation should answer six questions.
What is the claim?
The exact wording must be defined. “Secure”, “compliant” and “validated” are not testable without boundaries and criteria.
What is the object?
The assessor must identify the legal entity, product, version, model, module, service, location, operating environment and assessment period.
What are the criteria?
The applicable law, standard, specification, programme rule or contractual requirement must be identified, including its version.
What evidence supports the claim?
Evidence should be authentic, complete, current and traceable. Policies may prove intent. Logs, configurations, source records, test results and operational observations demonstrate implementation and performance.
Who performed the assessment?
The assessor’s competence, independence, authority and conflicts must be understood.
What could invalidate the result?
Material product changes, software releases, new environments, supplier changes, model drift, incidents and expired certificates may require revalidation.
A report that answers only whether something passed is incomplete. Decision-makers also need limitations, exclusions, unresolved findings, conditions of use and residual risk.
The board question
The board does not need to repeat the technical assessment.
It needs to know whether the organisation has relied on an interested party’s assertion or obtained independent evidence.
The questions are direct:
What claim are we relying upon?
What happens if it is wrong?
Who independently tested it?
Against which criteria?
What was excluded?
Which findings remain open?
Who owns the residual risk?
What change would invalidate the conclusion?
If management cannot answer those questions, it does not have assurance. It has confidence without sufficient evidence.
From assertion to evidence
Independent validation is not a substitute for competent engineering, compliance, certification or management accountability.
It is the control that tests whether their outputs deserve reliance.
In banking, it protects capital and risk decisions.
In cryptography, it separates FIPS claims from validated modules.
In cloud services, it tests whether controls operate within the service being purchased.
In product development, it challenges readiness before customers, regulators or investors expose the gaps.
In governance, it gives decision-makers evidence that did not originate solely from the people seeking approval.
The commercial value is straightforward.
A claim carries limited weight when the claimant is also the judge.
Independent validation changes that.
SITG-Consulting provides independent, evidence-led reviews of technology products, cryptographic claims, governance structures and operational readiness. The approach is structurally separate from product sales, equity participation and downstream implementation incentives.
Learn more:
Sources
Basel Committee on Banking Supervision, Internal Models Approach: General Provisions.bis
Bank of England Prudential Regulation Authority, SS1/23 Model Risk Management Principles for Banks.bankofengland.co
NIST, Cryptographic Module Validation Program FAQ.csrc.nist
NIST, Cryptographic Module Validation Program.nist
ISO, Conformity Assessment.iso
ISO, Certification and Accreditation.iso
PCAOB, Auditor Independence Spotlight.assets.pcaobus
US Securities and Exchange Commission, Knight Capital Americas administrative order.sec
Financial Conduct Authority, TSB operational resilience enforcement action.fca
NIST, Economic Impacts of Inadequate Infrastructure for Software Testing.
Comments