@formscript/lexer

Source text → tokens, per §3 of the specification.

Usage

import { tokenize } from '@formscript/lexer';

const { tokens, diagnostics } = tokenize(source, { file: 'application.forms' });

tokenize never throws. Every problem it finds is a diagnostic:

if (diagnostics.length > 0) {
  for (const d of diagnostics) {
    console.error(`${d.code} ${d.location?.line}:${d.location?.column} ${d.message}`);
  }
}

What it handles

• UTF-8 input, with a byte offset on every span.
• All three comment forms of §3.3 (#, //, /* … */).
• Identifiers, the hard keywords of §3.5 and the soft keywords of §20.
• Every literal kind of §3.6: integers, decimals, strings with the full escape set, booleans, null, inapplicable, dates, datetimes and durations.
• The punctuation of §3.7, longest-match first.

Notes

• A leading - is punctuation, not part of a numeric literal. The grammar's unary production handles the sign, so -1 and a - 1 are the same token stream.
• The duration literal follows the ISO 8601 form of §3.6, including P1Y2M3D and PT30M, and the short form (30d, 6mo, 2y). When a short form ends in m and is followed by o, the longer unit wins.

Public API

22 exported symbols.