libraryDocs / utf32

utf32

librarysharedsource

The utf32 library provides functions for the manipulation of UTF-32 strings.

Positions are in bytes, except utf32.sub, which takes code point positions. Code units are aligned to the start of the string, all functions use little-endian byte order unless big_endian is true, a byte order mark is not handled.

Functions that take lax raise an error on the first invalid code unit (a surrogate, a value above U+10FFFF or a truncated tail), unless lax is true; in lax mode each invalid code unit counts as one code point, decoded as U+FFFD.

Functions

`utf32.char`	Encodes code points into a little-endian UTF-32 string.
`utf32.codepoint`	Returns the code points of all characters that start between byte positions start_position and end_position (both inclu…
`utf32.codes`	Returns an iterator function that iterates over the code points of a UTF-32 string.
`utf32.decode`	Decodes a single UTF-32 sequence starting at the given byte position.
`utf32.encode`	Encodes a single code point into a UTF-32 sequence of 1 code unit (4 bytes).
`utf32.len`	Returns the number of code points that start between byte positions start_position and end_position (both inclusive).
`utf32.normalize`	Returns a well-formed copy of a UTF-32 string, every invalid sequence (surrogates, values above U+10FFFF, a truncated t…
`utf32.offset`	Returns the byte positions where the n-th code point, counting from byte position, starts and ends.
`utf32.pack`	Encodes a table of code points into a UTF-32 string.
`utf32.reverse`	Returns the code points between byte positions start_position and end_position in reverse order.
`utf32.sub`	Returns the substring between code points start_position and end_position (both inclusive).
`utf32.unpack`	Decodes a UTF-32 string into a table of code points.

#utf32.char

sharednodiscardsource

function utf32.char(a?: integer, b?: integer, ...: integer): string

Encodes code points into a little-endian UTF-32 string.

This functions similarly to string.char, invalid code points raise an error, use utf32.pack for big-endian strings.

Parameters

Name	Type	Description
`a`optional	`integer`	The first code point.
`b`optional	`integer`	The second code point.
`...`	`integer`	The rest of the code points.

Returns

Type	Name	Description
`string`	`utf32_string`	The UTF-32 string.

#utf32.codepoint

sharednodiscardsource

function utf32.codepoint(utf32_string: string, start_position?: integer, end_position?: integer, lax?: boolean, big_endian?: boolean): integer

Returns the code points of all characters that start between byte positions start_position and end_position (both inclusive).

This functions similarly to string.byte, use utf32.unpack for long strings.

The last code point is decoded completely even if it ends after end_position.

Parameters

Name	Type	Description
`utf32_string`	`string`	The UTF-32 string to decode.
`start_position`optional	`integer`	The position to start from in bytes, `1` by default.
`end_position`optional	`integer`	The position to end at in bytes, `start_position` by default.
`lax`optional	`boolean`	If `true`, invalid sequences are decoded as U+FFFD instead of raising an error.
`big_endian`optional	`boolean`	Whether the string is big-endian.

Returns

Type	Name	Description
`integer`	`...`	The code points.

#utf32.codes

sharednodiscardsource

function utf32.codes(utf32_string: string, lax?: boolean, big_endian?: boolean): ( fun( utf32_string: string, index: integer ): integer | nil, integer | nil ), string, integer

Returns an iterator function that iterates over the code points of a UTF-32 string.

Yields the byte position and the code point of every character, like utf8.codes.

Invalid sequences raise an error, or yield U+FFFD if lax is true.

Parameters

Name	Type	Description
`utf32_string`	`string`	The UTF-32 string to iterate over.
`lax`optional	`boolean`	If `true`, invalid sequences are yielded as U+FFFD instead of raising an error.
`big_endian`optional	`boolean`	Whether the string is big-endian.

Returns

Type	Name	Description
`( fun( utf32_string: string, index: integer ): integer \| nil, integer \| nil )`
`string`
`integer`

#utf32.decode

sharednodiscardsource

function utf32.decode(utf32_string: string, index?: integer, end_index?: integer, lax?: boolean, big_endian?: boolean, stack_level?: integer): integer | nil, integer, integer | nil, utf32.ErrorCode | nil

Decodes a single UTF-32 sequence starting at the given byte position.

In lax mode invalid sequences do not raise an error, nil is returned as the code point together with the number of bytes to skip, the position of the sequence and the error code.

Parameters

Name	Type	Description
`utf32_string`	`string`	The UTF-32 string to decode from.
`index`optional	`integer`	The position of the sequence in bytes, `1` by default.
`end_index`optional	`integer`	The last byte that can be read, the end of the string by default; a sequence cut by it is invalid.
`lax`optional	`boolean`	If `true`, invalid sequences are returned as `nil` instead of raising an error.
`big_endian`optional	`boolean`	Whether the string is big-endian.
`stack_level`optional	`integer`	The error level, relative to the caller, `1` (default) blames the caller of this function.

Returns

Type	Name	Description
`integer \| nil`	`utf32_codepoint`	The decoded code point, or `nil` if the sequence is invalid.
`integer`	`sequence_length`	The length of the sequence in bytes, at least `1`.
`integer \| nil`	`error_position`	The position of the invalid sequence, `nil` if the sequence is valid.
`utf32.ErrorCode \| nil`	`error_code`	The error code, `nil` if the sequence is valid.

#utf32.encode

sharednodiscardsource

function utf32.encode(utf32_codepoint: integer, lax?: boolean, big_endian?: boolean, stack_level?: integer): string

Encodes a single code point into a UTF-32 sequence of 1 code unit (4 bytes).

Parameters

Name	Type	Description
`utf32_codepoint`	`integer`	The code point to encode.
`lax`optional	`boolean`	If `true`, invalid code points (surrogates, negative or above `U+10FFFF`) are encoded as `U+FFFD` instead of raising an error.
`big_endian`optional	`boolean`	Whether to encode as big-endian.
`stack_level`optional	`integer`	The error level, relative to the caller, `1` (default) blames the caller of this function.

Returns

Type	Name	Description
`string`	`utf32_sequence`	The UTF-32 sequence.

#utf32.len

sharednodiscardsource

function utf32.len(utf32_string: string, start_position?: integer, end_position?: integer, lax?: boolean, big_endian?: boolean): integer

Returns the number of code points that start between byte positions start_position and end_position (both inclusive).

Parameters

Name	Type	Description
`utf32_string`	`string`	The UTF-32 string to count the code points of.
`start_position`optional	`integer`	The position to start from in bytes, `1` by default.
`end_position`optional	`integer`	The position to end at in bytes, the end of the string by default; a sequence cut by it is invalid.
`lax`optional	`boolean`	If `true`, every invalid sequence counts as one code point instead of raising an error.
`big_endian`optional	`boolean`	Whether the string is big-endian.

Returns

Type	Name	Description
`integer`	`utf32_codepoint_count`	The number of code points.

#utf32.normalize

sharednodiscardsource

function utf32.normalize(utf32_string: string, replacement_str?: string, start_position?: integer, end_position?: integer, big_endian?: boolean): string

Returns a well-formed copy of a UTF-32 string, every invalid sequence (surrogates, values above U+10FFFF, a truncated tail) is replaced with replacement_str.

This is not Unicode normalization (NFC, NFD, NFKC, NFKD): valid sequences are copied as is.

Only the range between start_position and end_position is returned, a sequence cut by end_position is invalid.

Parameters

Name	Type	Description
`utf32_string`	`string`	The UTF-32 string to sanitize.
`replacement_str`optional	`string`	The string to replace invalid sequences with, U+FFFD in the string's byte order by default.
`start_position`optional	`integer`	The position to start from in bytes, `1` by default.
`end_position`optional	`integer`	The position to end at in bytes, the end of the string by default.
`big_endian`optional	`boolean`	Whether the string is big-endian.

Returns

Type	Name	Description
`string`	`utf32_normalized`	The well-formed UTF-32 string.

#utf32.offset

sharednodiscardsource

function utf32.offset(utf32_string: string, n: integer, position?: integer, lax?: boolean, big_endian?: boolean): integer | nil, integer | nil

Returns the byte positions where the n-th code point, counting from byte position, starts and ends.

This functions similarly to utf8.offset from Lua 5.5:

n > 0 counts forward, 1 is the code point that starts at position.
n < 0 counts backward, -1 is the code point right before position.
n = 0 returns the code point that contains byte position.

#utf32_string + 1 is the position right after the last code point and is returned as #utf32_string + 1, #utf32_string + 1.

Code units are aligned to the start of the string, a truncated tail (1 to 3 bytes) is one code point. In lax mode positions are computed without reading the string, otherwise any invalid code unit that is walked over raises an error.

Raises an error if n ~= 0 and position points inside a code point.

Parameters

Name	Type	Description
`utf32_string`	`string`	The UTF-32 string to search in.
`n`	`integer`	The number of the code point, relative to `position`.
`position`optional	`integer`	The byte position to count from, `1` if `n >= 0`, otherwise `#utf32_string + 1`, negative positions count from the end of the string, out of range positions are clamped.
`lax`optional	`boolean`	If `true`, invalid code units count as code points instead of raising an error.
`big_endian`optional	`boolean`	Whether the string is big-endian.

Returns

Type	Name	Description
`integer \| nil`	`start_position`	The position of the first byte of the code point, or `nil` if the string has no such code point.
`integer \| nil`	`end_position`	The position of the last byte of the code point, or `nil` if the string has no such code point.

#utf32.pack

sharednodiscardsource

function utf32.pack(utf32_codepoints: integer[], utf32_codepoint_count?: integer, lax?: boolean, big_endian?: boolean): string

Encodes a table of code points into a UTF-32 string.

Parameters

Name	Type	Description
`utf32_codepoints`	`integer[]`	The code points to encode.
`utf32_codepoint_count`optional	`integer`	The number of code points to encode, `#utf32_codepoints` by default.
`lax`optional	`boolean`	If `true`, invalid code points (surrogates, negative or above U+10FFFF) are encoded as U+FFFD instead of raising an error.
`big_endian`optional	`boolean`	Whether to produce a big-endian string.

Returns

Type	Name	Description
`string`	`utf32_string`	The UTF-32 string.

#utf32.reverse

sharednodiscardsource

function utf32.reverse(utf32_string: string, start_position?: integer, end_position?: integer, lax?: boolean, big_endian?: boolean): string

Returns the code points between byte positions start_position and end_position in reverse order.

This functions similarly to string.reverse, but the bytes of each code point keep their order.

Parameters

Name	Type	Description
`utf32_string`	`string`	The UTF-32 string to reverse.
`start_position`optional	`integer`	The position to start from in bytes, `1` by default.
`end_position`optional	`integer`	The position to end at in bytes, the end of the string by default; a sequence cut by it is invalid.
`lax`optional	`boolean`	If `true`, invalid sequences are replaced with U+FFFD instead of raising an error.
`big_endian`optional	`boolean`	Whether the string is big-endian.

Returns

Type	Name	Description
`string`	`utf32_reversed`	The reversed UTF-32 string.

#utf32.sub

sharednodiscardsource

function utf32.sub(utf32_string: string, start_position?: integer, end_position?: integer, lax?: boolean, big_endian?: boolean): string

Returns the substring between code points start_position and end_position (both inclusive).

This functions similarly to string.sub, but positions are in code points, negative positions count from the end of the string.

The bytes are copied as is, in lax mode invalid sequences are kept (each counts as one code point).

Parameters

Name	Type	Description
`utf32_string`	`string`	The UTF-32 string to get the substring of.
`start_position`optional	`integer`	The position of the first code point, `1` by default.
`end_position`optional	`integer`	The position of the last code point, the last code point of the string by default.
`lax`optional	`boolean`	If `true`, every invalid sequence counts as one code point instead of raising an error.
`big_endian`optional	`boolean`	Whether the string is big-endian.

Returns

Type	Name	Description
`string`	`utf32_sub`	The substring.

#utf32.unpack

sharednodiscardsource

function utf32.unpack(utf32_string: string, start_position?: integer, end_position?: integer, lax?: boolean, big_endian?: boolean): integer[], integer

Decodes a UTF-32 string into a table of code points.

Parameters

Name	Type	Description
`utf32_string`	`string`	The UTF-32 string to decode.
`start_position`optional	`integer`	The position to start from in bytes, `1` by default.
`end_position`optional	`integer`	The position to end at in bytes, the end of the string by default; a sequence cut by it is invalid.
`lax`optional	`boolean`	If `true`, invalid sequences are decoded as U+FFFD instead of raising an error.
`big_endian`optional	`boolean`	Whether the string is big-endian.

Returns

Type	Name	Description
`integer[]`	`utf32_codepoints`	The code points.
`integer`	`utf32_codepoint_count`	The number of code points.