Table of Contents

1 Introduction

The regulatory trend toward increasingly granular data collection has highlighted performance limitations in traditional XBRL validation approaches. While xBRL-CSV reports handle large data volumes effectively (see xBRL-CSV tutorial and examples), validating data correctness using existing methods can become a significant processing bottleneck. The new draft xBRL-CSV Table Constraints specification exploits the structure of CSV and addresses some of these challenges by introducing efficient, built-in validation mechanisms that operate within the XBRL-CSV metadata framework.

This guidance document is targeted toward taxonomy authors, data collectors, and anyone wishing to understand the new specification. It provides an overview of the draft Table Constraints specification and outlines the key new features it introduces.

2 Validation Performance Challenges in XBRL Formula Rules

Modern regulatory requirements frequently involve substantial datasets. A commercial bank's quarterly liability reporting may include millions of individual records, each requiring validation of multiple mandatory and optional data points. Insurance companies submit policy-level data encompassing their entire portfolios. These datasets often exceed the practical limits of traditional validation approaches.

Current validation typically relies on XBRL Formula Rules, which face inherent scalability challenges when processing large open tables with typed dimensions. For datasets containing millions of rows, XBRL Formula rules based validation can require hours of processing time and substantial memory resources, often resulting in system timeouts or memory exhaustion.

The performance challenges arise from the computational demands of validating very large datasets, particularly those with millions of rows using typed taxonomy-defined dimension. While XBRL Formula Rules provide sophisticated validation capabilities through its filtering and implicit matching mechanisms, these features become resource-intensive when applied to substantial datasets.

Consider liability reporting where each liability is identified by a typed taxonomy-defined dimension. To validate completeness requirements (ensuring all mandatory facts are present for each liability), XBRL Formula Rules must:

1. Process dimensional matching across potentially millions of facts.
2. Apply implicit filtering to match facts by multiple aspects (period, entity, dimensions).
3. Evaluate complex expressions for each matched fact combination.
4. Maintain processing state across large fact collections¹.

While XBRL Formula Rule processors can optimise common filter expressions and variable binding operations, the fundamental challenge remains; as dataset size grows, the computational complexity of dimensional matching and aspect-based filtering scales significantly. Memory requirements increase substantially when processing large collections of facts that must be kept accessible for cross-referencing and validation.

3 xBRL-CSV Table Constraints: Validation Framework

xBRL-CSV Table Constraints embeds validation rules directly within xBRL-CSV metadata in JSON format. By defining constraints at the column and key levels, it shifts validation closer to the data itself, enabling faster, more precise checks during data ingestion. These xBRL-CSV Table Constraints support two major categories of validation:

3.1 Column-Level Validation

Constraints can be applied to individual columns to validate data types, value ranges, and format requirements, for example:

• Data Type Constraints: Validate against XML Schema built-in types or xBRL-CSV core dimensions (concept, entity, period, unit, language), as specified in the xBRL-CSV specification.
• Value Enumeration: Restrict values to a specific allowed set.
• Pattern Matching: Validate values against regular expressions.
• Mandatory Values: Specify columns that must contain a value for every row.
• Period-Specific Constraints: For period dimensions, constrain duration types (year, quarter, month, day, instant) or define custom durations.
• Time zone Validation: For date/time values, enforce presence or absence of time zone information.
• Duration Type Constraints: For XML Schema duration values, specify whether durations must be year-month based, or daytime based.

3.2 Key-Based Integrity Enforcement

Database-style key constraints can be defined to ensure data uniqueness, consistency and referential integrity between different table templates. These constraints include:

• Unique Keys: Establish and enforce unique row identification.
• Reference Keys: Validate relationships between tables.

For example, consider a regulatory submission with separate tables for individual customers and corporate customers, plus a reference table of valid country codes. xBRL-CSV Table Constraints can enforce that customer IDs are unique across both individual and corporate tables, while validating that any country codes referenced in corporate customer records actually exist in the countries reference table – all without loading complete datasets into memory.

4 xBRL-CSV Table Constraints and XBRL Formula Rules

xBRL-CSV Table Constraints functions in conjunction with existing XBRL Formula Rules validation rather than serving as a replacement. The two approaches serve complementary roles in a comprehensive validation strategy. xBRL-CSV Table Constraints excel at catching fundamental technical structural errors quickly and efficiently, while XBRL Formula Rules provides sophisticated validation capabilities (more details here: XBRL Formula Rules Tutorial) that require full taxonomy context.

5 xBRL-CSV Table Constraints: Benefits

xBRL-CSV Table Constraints help overcome some of the challenges seen with XBRL Formula Rules, especially around performance and early error detection. They add an efficient layer of validation that works well for large or complex datasets.

5.1 Streaming Validation

xBRL-CSV Table Constraints support streaming data processing which means the system can validate data in small, incremental chunks as it arrives—without waiting for the full file to be loaded, enabling:

• Constant memory operation: Process arbitrarily large files with minimal memory overhead.
• Parallel validation: Execute multiple constraint checks simultaneously.
• Immediate error detection: Identify issues during data ingestion rather than post-processing.

5.2 Performance Benefits

Organisations avoid expending computational resources on XBRL Formula Rules validation when fundamental data quality issues would cause that validation to fail anyway. A dataset with basic formatting errors or missing required fields can be rejected in minutes rather than hours.

5.3 Error Prioritisation

Sequencing xBRL-CSV Table Constraints and XBRL Formula Rules validations enables effective error prioritisation.

• Stage 1 - Table Constraints: Perform lightweight technical validation during data ingestion. If technical errors are detected, processing can halt immediately without proceeding to resource-intensive XBRL Formula Rule validation.
• Stage 2 - XBRL Formula Rule: Apply comprehensive business logic validation only after confirming basic data integrity.

Problems are identified and reported first, allowing data preparers to address fundamental issues before tackling more complex business rule violations. This staged approach optimises resource utilization while maintaining comprehensive validation coverage. Table Constraints handle what they do best – fast, efficient technical validation – while preserving XBRL Formula Rule's strength in complex business logic enforcement.

5.4 Cross-Table Validation

Uniqueness, consistency, and referential integrity across tables can be validated efficiently with xBRL-CSV Table Constraints, addressing scenarios that were previously challenging to implement using XBRL Formula Rules.

6 Backward Compatibility and Migration

xBRL-CSV Table Constraints extend xBRL-CSV 1.0 through backward-compatible metadata additions. Existing xBRL-CSV processors continue to function normally, while processors supporting xBRL-CSV Table Constraints provide enhanced validation capabilities. This compatibility enables:

• Incremental adoption without disrupting existing workflows.
• Gradual migration from XBRL Formula Rule based validation.
• Coexistence of multiple validation approaches during transition periods.

7 Specification Status

xBRL-CSV Table Constraints 1.0 has achieved Candidate Recommendation status, indicating readiness for implementation and testing. The specification provides comprehensive guidance for constraint definition, validation processing, and error reporting. Early implementations demonstrate the practical feasibility of the approach and validate the performance improvements anticipated during specification development.

8 Conclusion

xBRL-CSV Table Constraints represent a significant advancement in xBRL-CSV report processing capability, directly addressing the scalability challenges that have emerged as regulatory reporting requirements have evolved toward greater granularity. By embedding validation logic within the data structure itself, the specification enables efficient processing of large datasets while maintaining the data quality standards essential for regulatory and business reporting. The specification provides a clear path forward for handling increasing data volumes without compromising validation thoroughness or system performance.