@formscript/conformance

The conformance suite of the FormScript Language Specification 1.0.0 , §18 — and the runner that executes it.

§18 is not a description of a suite; it is the suite, written in the specification. This package extracts those rows from the specification text and runs them against the reference implementation, so the claim "the suite is executed, not asserted" is a fact with a test result behind it.

The five sections

Two rows that are easy to get wrong

§18.3 distinguishes result type from result value. The row 1 + 1 → 2 and the row 1 + 1.0 → 2.0 have the same value and different types. The runner therefore compares the value with strictEquals and the static type with the analyzer's inferType. A runner that only checked the value would pass a runtime that computed integers where decimals were required.

§18.5 asserts the keys that must be absent. The flagship's output document omits an inapplicable entity, so the expected result names postcode, guarantor.guarantor_name and guarantor.guarantor_consent as absent (Spec-4.35). A runtime that emitted an inapplicable field as null would fail this row.

The flagship's inputs are raw text

FLAGSHIP_ASSIGNMENTS assigns '30000', not 30000. The engine rejects a JavaScript double for a number field, which is correct for a decimal data plane: the input of a form is text, and Spec-5.3 governs its coercion.

Public API

const suite = readSuite('FormScript-Language-Specification.md');
const report = runSuite(suite);

report.passed;    // 101
report.failed;    // 0
report.totals;    // { '18.1': { passed: 46, failed: 0 }, ... }
report.failures;  // one entry per failing row

Also exported: extractSuite, parseEvaluationTable, parseRejectTable, splitDocuments, diagnose, reportedDiagnostics, literalValue, runEvaluationCase, runFlagship, flattenOutput, FLAGSHIP_ASSIGNMENTS, FLAGSHIP_OUTPUT, FLAGSHIP_ABSENT_KEYS and FLAGSHIP_OUTPUT_KEYS.

Running it

npm run conformance
# or, through the command line, with the same output:
npm run formscript -- conformance

The runner reports each section, then the failures with the expected and the actual result, and exits non-zero if any row failed.

Judgement calls

1. The rows are read from the specification, not duplicated here. A row added to §18.3 of the specification is picked up by the next run; there is no second copy to keep in step.
2. §18.2 checks that the expected code is among the reported codes, not that it is the only one. The specification's rows each isolate a defect, but a document that also trips an unrelated rule still demonstrates the behaviour the row claims to test.
3. The stage order of Spec-2.5 is applied to the §18.2 rows. The reported set is the errors of the earliest non-empty stage plus every warning, so a lexical error is not masked by a semantic one further down.
4. The §18.5 fixture is the §17.2 document verbatim. It is extracted from the specification, so the flagship that is tested is the flagship that is published.

Public API

33 exported symbols.