tegral-niwen-lexer/guru.zoroark.tegral.niwen.lexer/Token

Token

data class Token(val string: String, val startsAt: Int, val endsAt: Int, val tokenType: TokenType)

A Token is a data class representing a token found through the tokenization process of a Lexer and the matching process of a TokenMatcher. It has information on where the token begins, where it ends, its type, and what it actually represents.

Constructors

Token

constructor(string: String, startsAt: Int, endsAt: Int, tokenType: TokenType)

Properties

endsAt

val endsAt: Int

The index after string ends in the original string (i.e. exclusive).

startsAt

val startsAt: Int

The index where string starts in the original string.

string

val string: String

The string this token represents. This is exactly what was match and, as such, is a substring of the original lexed string

tokenType

val tokenType: TokenType

The token type this token corresponds to