Lexer

data class Lexer(states: Map<StateLabel?, LexerState>, defaultStateLabel: StateLabel? = null)

A Lexer is a data class that contains the states that will be used for the lexing process. It is the main entry for using a constructed lexer.

The class itself only contains states and not much else.

The function tokenize can be used to use the lexer on a string

Constructors

constructor(states: Map<StateLabel?, LexerState>, defaultStateLabel: StateLabel? = null)

Get the state with the given label

The tokenize method will turn a string into a list of tokens based on the LexerStates contained in this Lexer (states) and the

The default state in the case of a multi labeled state lexer, or the only state of the lexer in the case of a single unlabeled state lexer

The amount of states contained within this lexer