Formula and related specifications:
Proposed Recommendation Overview
Recommendation dated 2009-06-22
This document:
	XBRL-Formulas-Overview-REC-2009-06-22.rtf
is non-normative.
Editors
Geoff Shuetrim	geoff@galexy.net	Galexy	
Contributors

Paul Bull	paul.bull@morganstanley.com	Morgan Stanley	
Jim Richards	jdrassoc@iinet.com.au	JDR & Associates	
Herm Fischer	fischer@markv.com	UBmatrix & Mark V Systems	
Abstract
This document explains the various materials issued as a part of the proposed recommendation released by the Formula Working Group (FWG). It has been augmented to incorporate the generic link specification that provides the basic syntax upon which the formula and related specifications build. Along with the specifications and related documents provided with this candidate recommendation, this overview addresses the specific requirements that need to be met by candidate recommendations.
Status
Circulation of this Proposed Recommendation is unrestricted.  Recipients are invited to email comments to formula-feedback@xbrl.org, and to submit notification of any relevant patent rights of which they are aware and to provide supporting documentation.

Table of Contents
1Introduction	2
2Relationship to existing international standards and related work outside of XBRL International	3
3Dependencies with other working groups	4
4Formal objections to the specifications	4
5Requirements	4
6Specifications	5
7Schemas	7
8Examples	7
9Conformance suite tests	7
10Function registry	7
11Changes since the previous CR	8
A.References	10
B.Intellectual Property Status	11
C.Document history	12
D.Approval process	13

1	Introduction
This document provides an overview of the Proposed Recommendation (PR) of the formula and related specifications, including the generic link specification.  It describes the various materials included in the PR.
The PR includes:
•	the requirements agreed by the FWG for formula and for XBRL instance validation against business rules;
•	the generic link requirements agreed by the Base Specification Working Group (BSWG), in consultation with other working groups;
•	each of the specifications, including normative schemas in appendices;
•	separate non-normative schemas for each of the specifications;
•	XBRL function definitions that support the implied XPath expressions [XPATH] in the formula and related specifications; and
•	a selection of examples.
A Public Working Draft (PWD) of the conformance suite for the formula and related specifications is published in conjunction with this PR.  It is being published as a PWD because of the expectation that it will require changes at a more rapid rate than the specifications themselves as implementation experience accumulates.
A Public Working Draft (PWD) of the function registry is also published in conjunction with this PR.
2	Relationship to existing international standards and related work outside of XBRL International
The XBRL Specification [XBRL] provides a syntax for business reports.  The XBRL Formula, Variable and Filter specifications provide a syntax for expressing rules that can be used t derive new fact values from the data in XBRL business report.  These rules are referred to as formulae.  The generic label and reference specifications support labelling of all manner of different XBRL constructs. In the context of XBRL formulae, they can be used to associate human documentation with formulae, their variables and the filters that define which facts in an XBRL business report get selected by a variable for usage in the evaluation of a formula. The validation and the three assertion specifications define a syntax for expressing rules about the expected content of business reports, in terms of variables, sets of variables and formulae.
Together these specifications are intended to support expression of rich formulaic relationships that can then be communicated between producers and consumers of XBRL business reports.
The formulae that can be expressed using the syntax defined in theses specifications bears some comparison to the formulae that can be expressed using spreadsheets and the various Specifications for expressing spreadsheets as XML (See the SpreadsheetML  [SML] and the Open Document Format [ODF] specifications).
The spreadsheet method of writing formulae relies on spreadsheet coordinates (row and column identifiers) to identify values that are used as inputs or outputs of a formula. Users often associate labels with those coordinates to communicate the meaning of the data in the range of cells identified by the coordinates.  Formulae are expressed as operations on values that are identified by the spreadsheet coordinates or the labels given to those coordinates.
In XBRL, a value in a business report is identifiable by the information associated with that value.  These include the concept that the value measures, the entity that the value describes, the period over which or at which the measurement was taken, the units in which the measurement is taken, and, potentially, a range of other aspects given more specificity in terms of the part of the entity being reported on and the scenario that should be recognised when interpreting the fact.  These aspect values are used in place of co-ordinates by XBRL formulae when specifying  their inputs and outputs.
This feature of XBRL formulae comes at a price. Specifically, an XBRL business report can contain more than one set of facts meeting a specified identification (in terms of a set of aspect values).  In such cases, the response of an XBRL formula processor will depend on the formula usage pattern.  Over time, it is expected that a growing number of usage patterns will be defined in extension specifications.  For now, the response of formula processors is application dependent.
The XBRL formula specification also addresses another constraint associated with the spreadsheet method of communicating formulas. In spreadsheets formulae must explicitly use all of the coordinates to identify each input. XBRL Formulae use implicit matching of aspect values. 
Implicit matching is best described by way of an example. The formula author defines ‘GrossProfit = Sales – CostOfGoodsSold’ in an XBRL taxonomy. This formula is expressed in terms of inputs that are defined by one aspect: the concept that is measured.  No other aspects are referred to in the formula and so all of the non-referenced aspects are matched across the variables used in the formula: in this case Sales and CostOfGoodsSold.  This means that the formula can be used to compute GrossProfit for any period, any entity and in any units so long as both Sales and CostOfGoodsSold are both reported with the same values for all of those aspects.
Implicit matching reduces the burden on the author of having to explicitly refer to all of the aspects and dimensions that could be encountered when a formula is processed. This is particularly important in XBRL, where a formula author may not know all of the aspects or aspect values that will be associated with facts that will be used as inputs to the formulae being authored.
The formula and related specifications also bear comparison to XQuery [XQUERY].  Xquery, especially when supported by a broad range of XBRL specific functions, can be used to express any of the formulae that can be expressed by the syntax in the XBRL formula and related specifications.  A crucial reason for developing a new syntax for XBRL formulae when XQuery is already available is that while XQuery programs implementing XBRL formulae may be excellent for formula execution purposes they do not, in general, provide transparent documentation of the relationships between the facts or the classes of facts that are described by formulae.  This documentation role for formulae is particularly important for XBRL where such documentation is crucial to how business choose which aspects to associate with the facts that they report.
The XBRL specification also bears comparison to MathML [MML].  XBRL formulae are considerably more mathematically limited than MathML because they express their formulae using XPath 2.0 [XPATH].  This choice was made in light of the relatively broad and complete  range of implementations of XPath 2.0 compared to MathML.
The validation and assertion specifications (which leverage the formula and related specifications) bear some relationship to Schematron [STRON] in that they support making verifiable statements about the content of XML documents.  XBRL assertions differ from Schematron assertions, in that they are specialised to XBRL whereas Schematron is a language for expressing assertions about patterns in a very broad range of XML documents.  Schematron is so widely useful because it operates well on patterns that are reflected in the hierarchical structure of XML documents.  This strength in Schematron is a handicap when used with XBRL because many of the important XBRL relationships are expressed using XLink [XLINK] rather than the XML tree structure.  XBRL assertions, by leveraging the aspect model that underpins formulae and variables, are able to be expressed succinctly and clearly while still being able to work with the XBRL relationships expressed using Xlink.
3	Dependencies with other working groups
There are no changes in the dependencies of these specifications on the product of other working groups.

(The implementations of processors for these specifications do depend on the products of the Base Spec Working Group, notably the XBRL 2.1 and Dimensions specifications.  However the formula specification is intended to be isolated from those specifications.  For example, although at time of this writing there is debate on Default Dimensions, the formula specification only addresses that such default dimensions are visible aspects of fact items for filtering and such, but does not tread into how a Dimensions compliant linkbase might specify them.)
 
Figure 1.  Dependencies on other standards
4	Formal objections to the specifications
No formal (or informal) objections have been raised in relation to the specifications since the previous CR was released.
5	Requirements
The requirements agreed upon by the FWG are included in the previous CR and remain unchanged.  There are requirements documents for:
•	formula requirements
•	XBRL instance business-rule validation requirements
•	generic linking requirements
The requirements have not been altered since their previous publication.
As for the previous CR, the only unfulfilled requirements relate to the ability to generate tuples using formula and the ability to generate fraction item types using formulae.
The ability to generate tuples is being addressed using an extension usage pattern specification.  Discussion of this extension usage pattern is posted on the working group wiki page for that subject. (Also see section 11 Extension Usage Patterns.)
The ability to generate output fraction facts may be addressed by a future specification that specifically targets output facts with fraction data types, if use cases emerge warranting such a feature.  The extension would involve a new formula-type resource with a specifically designed value production rule to cover numerators and denominators.
6	Specifications
The specifications of this PR are those of the prior CR with wording and clarification improvements, in regard to public comments, as discussed on the CR2 wiki page.  No semantic or strategic changes have occurred.
The generic linking specifications in this PR are:
•	Generic links (defining generic extended links that can be included in the meta-data supporting XBRL business reports)
•	Generic labels (defining labels that can be expressed within generic links)
•	Generic references (defining references that can be expressed within generic links)
The variable specification and supporting filter specifications are:
•	Variables (defining ways of selecting and transforming information from XBRL business reports)
•	Boolean filters (that allow for variable selection criteria that include AND and OR structures)
•	Concept filters (that allow for variable selection criteria based on concept information)
•	Dimension filters (that allow for variable selection criteria based on concept information)
•	Entity filters (that allow for variable selection criteria based on concept information)
•	General filters (that allow for variable selection criteria based on concept information)
•	Implicit filters (that allow for variable selection criteria based on concept information)
•	Match filters (that allow for variable selection criteria based on concept information)
•	Period filters (that allow for variable selection criteria based on concept information)
•	Relative filters (that allow for variable selection criteria based on concept information)
•	Segment-scenario filters (that allow for variable selection criteria based on concept information)
•	Tuple filters (that allow for variable selection criteria based on concept information)
•	Unit filters (that allow for variable selection criteria based on concept information)
•	Value filters (that allow for variable selection criteria based on concept information)
The modularisation of the filter specifications allows the variable specification to be extended with new filter specifications, should motivating use cases be identified.
There is a single formula specification that builds upon the generic link specification and the variables specification to define a syntax for XBRL formulae.
The validation specifications in this PR are:
•	Validation (defining the basic terms and structures that are common to all kinds of business rule validations of data in XBRL business reports)
•	Consistency assertions (defining testable assertions about the consistency of information in XBRL business reports with specific XBRL formulae)
•	Existence assertions (defining testable assertions about the existence of specific information in XBRL business reports)
•	Value assertions (defining testable assertions about the values of specific facts in XBRL business reports)
Some individuals may find it easier to read the generic link specification first, followed by the variable specification and some or all of the filter specifications. The formula specification and the validation/assertion specifications may then be more easily understood.
Note that the variable, filter and formula specifications are intended to form building blocks for more specialised usage patterns.  They enable expression of data selection and transformation rules and data generation rules.  In and of themselves, these are useful for XBRL data selection and XBRL data production. However, they can also be combined to perform more specialised operations, described within the FWG as usage patterns.

Figure 2.  Dependencies between specifications.
The PR only includes specifications to cover one such usage pattern: that being the XBRL report validation usage pattern.  Other specialised usage patterns for formulae and variables will be addressed in extension specifications while the formula and related specifications are developing implementation experience.  Among these usage patterns, priority will be given to tuple creation, and formula chaining.
7	Schemas
Normative schemas for the specifications in this PR are contained in appendices to those specifications.  Separate non-normative versions of the schemas supporting the specifications in this PR are provided as part of this PR.
8	Examples
The set of examples supporting this PR extend on the set of examples provided with the previous PR.
9	Conformance suite tests
A conformance suite for these specifications has been published as a PWD.  This contains a broad range of examples illustrating usage of the specifications.  Those with access to the FWG Subversion repository will be able to take advantage of the very latest versions of the test cases.
10	Changes since the previous CR
No changes have been made to the generic link, generic label or generic references specifications since they were last published aside from the addition of some convenience role and arcrole declarations to the normative schemas.
Some clarification refinements have been made to the formula and related specifications since the last CR.  The feedback motivating those refinements, the discussion and examples used by the FWG to analyse the feedback, and the eventual resolutions to the issues raised in the feedback are all documented extensively.
Additional explanations are provided in the revision histories contained by each of the documents.
Diff files generated for the underlying XML documents (as well as the current and previous XML documents) are provided for those wanting to review the exact modifications made to the specifications since their previous public release.
11	Extension Usage Patterns
There are a broad range of usage patterns for XBRL formulae.  Several of these usage patterns have been assessed as having highest priority among potential users of the formula and related specifications.
It is the intent that the usage patterns can be addressed by specifications that build upon the foundation provided by the formula and related specifications.  The FWG has undertaken a serious assessment of the highest priority usage patterns to determine whether they can be addressed by specifications that build upon the formula and related specifications without requiring changes to those foundation specifications.  The conclusion reached:
•	by assessing requirements for the usage patterns;
•	designing sketch specifications that address the usage patterns; and
•	drawing on findings from prototypes created by vendors,
is that the usage patterns can be addressed without requiring modifications to the formula and related specifications.
Formula Chaining
Formula chaining is a refinement of the iterative processing model where one output instance (augmented by the input data) becomes the next input instance.  It can be very important from a maintenance perspective to be able to modularise complex formulae into a sequence of simpler computations that can be chained together to achieve desired results. This kind of modularisation is inherent in spreadsheet applications. 
A = B + C, C = D + E
This is a use case suggested with two formulas, A = B + C, and C = D + E.  The formula chaining approach could process the C = D + E first, obtaining its result C then to use in the A = B + C formula.
An approach to this is being discussed on the working group wiki page, and there is general agreement in the FWG that the approach addresses requirements and does not require modifications to the formula and related specifications.
Both vendors of conformant formula processors are studying it, with an implemention of a prototype with UBmatrix XPE.
Tuple Generation
Tuple generation was an early requirement of formula that has become more of a step-child as most major XBRL implementations have banned use of tuples.  It is of great interest to the GL specifications, though, for output production of GL instance reports.  (The current specifications being proposed above are capable of tuple inputs, it is their output that is the extension usage pattern.)
An alternate approach proposed for formula chaining forms a basis for tuple generation, as it involves both the processing order (containing tuples before contained items and nested tuples), and the issue of location dependency (which tuple to contain which item and nested tuple).
Multi-instance processing
An early decision in producing these specifications was to not do multi-instance processing, at the start it was thought one could merge separate instances to a combined package for processing.  A specification covering how to merge XBRL instances and addressing situations where such merging operations are not possible will be able to address multi-instance processing without impacting on the formula and related specifications.
Assertion processing reports
Early assertion adopters require human readable messages to be associated with assertion results. Each assertion result is to imply a message to users where that message can contain information involved in the assertion testing.
Messages for assertion results have been implemented by both vendors for specific customer requirements, and a proposal is now on the wiki pages to evolve a standard common interoperable manner of specifying these messages.  No change to the formula specifications above was needed to implement this extension usage pattern.

A.	Reference

[MML]	Mathematical Markup Language (MathML) Version 2.0 (Second Edition)
W3C Recommendation 21 October 2003
http://www.w3.org/TR/2003/REC-MathML2-20031021/	
[ODF]	ISO/IEC 26300, 2006
Open Document Format for Office Applications 1.0
http://standards.iso.org/ittf/PubliclyAvailableStandards/index.html	
[SML]	SpreadsheetML
http://www.spreadsheetml.com/	
[STRON]	ISO/IEC 19757-3, 2006
Document Schema Definition Language (DSDL) -- Part 3: Rule-based validation -- Schematron
http://standards.iso.org/ittf/PubliclyAvailableStandards/index.html	
[XQUERY]	XQuery 1.0: An XML Query Language
W3C Recommendation 23 January 2007
http://www.w3.org/TR/xquery/	
[XBRL]	Extensible Business Reporting Language (XBRL) 2.1 Recommendation with corrected errata to 2006-12-18.
http://www.xbrl.org/SpecRecommendations/	
[XDT]	XBRL Dimensions 1.0
Recommendation, dated 2006-09-18
http://www.xbrl.org/Specification/XDT-REC-2006-09-18.rtf	
[XLINK]	XML Linking Language (XLink) Version 1.0
W3C Recommendation 27 June 2001
http://www.w3.org/TR/xlink/	

B.	Intellectual Property Status
This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works.  However, this document itself may not be modified in any way, such as by removing the copyright notice or references to XBRL International or XBRL organizations, except as required to translate it into languages other than English.  Members of XBRL International agree to grant certain licenses under the XBRL International Intellectual Property Policy (www.xbrl.org/legal).
This document and the information contained herein is provided on an "AS IS" basis and XBRL INTERNATIONAL DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
The attention of users of this document is directed to the possibility that compliance with or adoption of XBRL International specifications may require use of an invention covered by patent rights. XBRL International shall not be responsible for identifying patents for which a license may be required by any XBRL International specification, or for conducting legal inquiries into the legal validity or scope of those patents that are brought to its attention. XBRL International specifications are prospective and advisory only. Prospective users are responsible for protecting themselves against liability for infringement of patents.  XBRL International takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights.  Members of XBRL International agree to grant certain licenses under the XBRL International Intellectual Property Policy (www.xbrl.org/legal).

C.	Document history
Date	Editor	Summary	
2008-02-14	Geoff Shuetrim	First draft.	
2008-02-19	Geoff Shuetrim	Added section on relationship to other specifications as required for a last call for comments, drawing on extensive drafting by Paul Bull.	
2008-02-20	Geoff Shuetrim	Incorporated editorial feedback from Jim Richards.	
2008-03-12	Geoff Shuetrim	Updated for CR release.	
2008-03-13	Geoff Shuetrim	Incorporated feedback from CompSci resources.	
2008-11-03	Geoff Shuetrim	Updated for second CR release.  Merged in the necessary information from the generic link specification overview for its most  PWD.	
2009-03-26	Herm Fischer	Updated for PR release.	
2009-06-22	Herm Fischer	Updated for REC release.	
D.	Approval process
This document will not be released as a part of any final recommendation.