libraryDocs / lua / lexer

lua.lexer

librarysharedsource

Lua 5.1 / LuaJIT lexer.

Tokens are returned as flat arrays ( types, values, lines ) instead of token tables, whitespace, newlines and comments are kept as tokens, so concatenating all values gives back the original code.

Fields

Name	Type	Description
`TOKEN_SPACE`	`integer`	Spaces and tabs, without newlines. = `1`
`TOKEN_NEWLINE`	`integer`	A single `\n`. = `2`
`TOKEN_COMMENT`	`integer`	`--`, `--[[ ]]` comments and the shebang line. = `3`
`TOKEN_STRING`	`integer`	Quoted and long bracket strings, the value includes quotes / brackets. = `4`
`TOKEN_NUMBER`	`integer`	Number literals, including LuaJIT `LL`, `ULL` and `i` suffixes. = `5`
`TOKEN_NAME`	`integer`	Names and keywords. = `6`
`TOKEN_SYMBOL`	`integer`	Operators and punctuation. = `7`
`TOKEN_DIRECTIVE`	`integer`	A `#<name>` line accepted by the `is_directive` callback of `lexer.tokenize`, including `\` continued lines. = `8`
`TOKEN_NAMES`	`string[]`	Token type names, indexed by token type.
`keywords`	`table<string, boolean>`	Reserved words of Lua 5.1 and LuaJIT ( `goto` ). `continue` is not here, the parser treats it as a keyword only at the start of a statement.

Functions

`lua.lexer.countNewlines`	Counts \n characters in the string.
`lua.lexer.isIdentifier`	Checks if the string is a valid Lua name that is not a keyword.
`lua.lexer.tokenize`	Splits Lua code into tokens.

#lua.lexer.countNewlines

sharednodiscardsource

function lua.lexer.countNewlines(str: string): integer

Counts \n characters in the string.

Parameters

Name	Type	Description
`str`	`string`	The string to count newlines in.

Returns

Type	Name	Description
`integer`	`count`	The number of newlines.

#lua.lexer.isIdentifier

sharednodiscardsource

function lua.lexer.isIdentifier(str: string): boolean

Checks if the string is a valid Lua name that is not a keyword.

continue is rejected too, so the result is safe to use as a statement start.

Parameters

Name	Type	Description
`str`	`string`	The string to check.

Returns

Type	Name	Description
`boolean`	`is_identifier`	`true` if the string can be used as a variable or field name.

#lua.lexer.tokenize

sharedsource

function lua.lexer.tokenize(code: string, is_directive?: ( fun( name: string ): boolean )?): integer[], string[], integer[], integer

Splits Lua code into tokens.

Never fails, malformed tokens ( unfinished strings, invalid numbers ) are left for the parser to report.

Parameters

Name	Type	Description
`code`	`string`	The code to tokenize.
`is_directive`optional	`( fun( name: string ): boolean )?`	If set, `#<name>` at the start of a line produces a `TOKEN_DIRECTIVE` token when this function returns `true`.

Returns

Type	Name	Description
`integer[]`	`types`	The token types, see `lexer.TOKEN_*`.
`string[]`	`values`	The token source texts.
`integer[]`	`lines`	The lines where tokens start.
`integer`	`count`	The number of tokens.