tegral-niwen-lexer/guru.zoroark.tegral.niwen.lexer

Package-level declarations

Types

State labels created by stateLabel are instances of this class.

class GenericTokenType(val name: String) : TokenType

A generic class for token types. tokenType returns tokens of this type. Big lexers should be using an enum which implements TokenType instead of tokenType and GenericTokenType.

IgnoringMatcherBuilder

class IgnoringMatcherBuilder(val baseRecognizer: TokenRecognizer) : TokenMatcherBuilder

A DSL environment for constructing ignoring matchers. This builds into a TokenRecognizerIgnored.

Lexer

data class Lexer(states: Map<StateLabel?, LexerState>, defaultStateLabel: StateLabel? = null)

A Lexer is a data class that contains the states that will be used for the lexing process. It is the main entry for using a constructed lexer.

LexerBuilder

class LexerBuilder : Buildable<Lexer>

This class is used to build a Lexer using the Niwen Lexer DSL, and is used in lambda-with-receivers to provide high level DSLs for the configuration of a lexer.

LexerState

data class LexerState(val matchers: List<TokenMatcher>)

A state that contains matchers. The list of matchers is tested sequentially in order.

MatchedMatcherBuilder

class MatchedMatcherBuilder(val baseRecognizer: TokenRecognizer, matchesToTokenType: TokenType) : TokenMatcherBuilder

A simple builder for matchers, whose main purpose is to provide a way to select a "next state" for a matcher through the thenState function.

NiwenLexerException

open class NiwenLexerException(message: String, cause: Exception? = null) : TegralException

Generic parent class for exceptions thrown within the Niwen Lexer system

NiwenLexerNoMatchException

class NiwenLexerNoMatchException(message: String) : NiwenLexerException

This exception signals that no match were found for a substring for the current state.

StateBuilder

class StateBuilder : Buildable<LexerState>

This classed is used to build a lexer state (LexerState) using a DSL, and is used inside a lambda-with receiver. It also defines high level functions for constructing matchers, such as anyOf or matches, and using custom matchers with the + operator.

StateLabel

interface StateLabel

Represents a label for a state in a multiple labeled state context. Can be implemented by anything.

Token

data class Token(val string: String, val startsAt: Int, val endsAt: Int, val tokenType: TokenType)

A Token is a data class representing a token found through the tokenization process of a Lexer and the matching process of a TokenMatcher. It has information on where the token begins, where it ends, its type, and what it actually represents.

TokenMatcherBuilder

abstract class TokenMatcherBuilder(val baseRecognizer: TokenRecognizer) : Buildable<TokenMatcher>

An abstract builder for token matchers. This is used for supporting .ignored, thenState, etc.

TokenType

interface TokenType

A token type. Can be pretty much anything.

Functions

matcher

fun matcher(nextState: NextStateBehavior = NoStateChange, matcherBody: (s: String, startAt: Int) -> Token?): TokenMatcher

Simple function to create a matcher. The returned TokenMatcher executes the given lambda and returns its value and nothing else.

niwenLexer

fun niwenLexer(body: LexerBuilder.() -> Unit): Lexer

Creates a Lexer object using the Niwen Lexer DSL, where body receives a LexerBuilder that can be used to modify the lexer that is eventually returned.

stateLabel

fun stateLabel(): GenericStateLabel

Create a generic state label (actually an instance of GenericStateLabel and returns it. This function provides an easy and dirty way of creating labels when you do not have an enum in place.

tokenType

fun tokenType(name: String = ""): GenericTokenType

Creates a new, distinct token type and returns it. The returned token type is of the type GenericTokenType