TTMSFNCMemoCustomLanguageTokenizerItem Class

A named tokenizer state containing an ordered list of rules; the tokenizer evaluates the active state's rules against the input stream during lexing.

API unit family: TMSFNCMemoCustomLanguage Inherits from: TCollectionItem

Syntax

TTMSFNCMemoCustomLanguageTokenizerItem = class(TCollectionItem)

Properties

Name	Description
Name	The unique name of this tokenizer state (e.g. root, string, comment).
Rules	The ordered list of tokenizer rules evaluated when this state is active.

Methods

Name	Description
AsJSObjectProperty	Serialises this tokenizer state as a named JSON property containing its rule array.