Module toml.scanner stdlib

toml.scanner
Version:
0.3.3
License:
MIT
Dependencies from vmod:
0
Imports:
3
Imported by:
4
Repository:
OS-specific
Show selected OS-specific symbols.
Backend-specific
Show selected Backend-specific symbols.

Dependencies defined in v.mod

This section is empty.

Imports

Imported by

Aliases

This section is empty.

Constants

#constant digit_extras

digit_extras = [`_`, `.`, `x`, `o`, `b`, `e`, `E`]

#constant end_of_text

end_of_text  = u32(~0)

Sum types

This section is empty.

Functions

#fn new_simple

fn new_simple(config Config) !Scanner

new_simple returns a new stack allocated Scanner instance.

#fn new_simple_file

fn new_simple_file(path string) !Scanner

new_simple_file returns a new stack allocated Scanner instance ready for parsing TOML in file read from path.

#fn new_simple_text

fn new_simple_text(text string) !Scanner

new_simple_text returns a new stack allocated Scanner instance ready for parsing TOML in text.

Structs

#struct Scanner

pub struct Scanner {
pub:
	config Config
	text   string // the input TOML text
mut:
	col        int // current column number (x coordinate)
	line_nr    int = 1 // current line number (y coordinate)
	pos        int // current flat/index position in the `text` field
	header_len int // Length, how many bytes of header was found
	// Quirks
	is_left_of_assign bool = true // indicates if the scanner is on the *left* side of an assignment
}

Scanner contains the necessary fields for the state of the scan process.

the task the scanner does is also refered to as "lexing" or "tokenizing".

The Scanner methods are based on much of the work in vlib/strings/textscanner.

#fn new_scanner

fn new_scanner(config Config) !&Scanner

new_scanner returns a new heap allocated Scanner instance, based on the file in config.input.file_path, or based on the text in config.input.text .

#fn (&Scanner) scan

direct_array_access
fn (mut s &Scanner) scan() !toml.token.Token

scan returns the next token from the input.

#fn (&Scanner) free

unsafe
fn (mut s &Scanner) free()

free frees all allocated resources.

#fn (&Scanner) remaining

inline
fn (s &Scanner) remaining() int

remaining returns how many characters remain in the text input.

#fn (&Scanner) next

direct_array_accessinline
fn (mut s &Scanner) next() u32

next returns the next character code from the input text.

next returns end_of_text if it can't reach the next character.

#fn (&Scanner) skip

inline
fn (mut s &Scanner) skip()

skip skips one character ahead.

#fn (&Scanner) skip_n

inline
fn (mut s &Scanner) skip_n(n int)

skip_n skips ahead n characters.

If the skip goes out of bounds from the length of Scanner.text, the scanner position will be sat to the last character possible.

#fn (&Scanner) at

direct_array_accessinline
fn (s &Scanner) at() u32

at returns the current character code from the input text.

at returns end_of_text if it can't get the current character.

unlike next, at does not change the state of the scanner.

#fn (Scanner) at_crlf

fn (s Scanner) at_crlf() bool

at_crlf returns true if the scanner is at a \r character and the next character is a \n.

#fn (&Scanner) peek

direct_array_accessinline
fn (s &Scanner) peek(n int) u32

peek returns the character code from the input text at position + n.

peek returns end_of_text if it can't peek n characters ahead.

#fn (&Scanner) reset

fn (mut s &Scanner) reset()

reset resets the internal state of the scanner.

#fn (&Scanner) new_token

inline
fn (mut s &Scanner) new_token(kind toml.token.Kind, lit string, len int) toml.token.Token

new_token returns a new token.Token.

#fn (&Scanner) ignore_line

direct_array_accessinline
fn (mut s &Scanner) ignore_line() !string

ignore_line forwards the scanner to the end of the current line.

#fn (&Scanner) inc_line_number

inline
fn (mut s &Scanner) inc_line_number()

inc_line_number increases the internal line number.

#fn (&Scanner) extract_key

direct_array_accessinline
fn (mut s &Scanner) extract_key() string

extract_key parses and returns a TOML key as a string.

#fn (&Scanner) extract_string

direct_array_accessinline
fn (mut s &Scanner) extract_string() !string

extract_string collects and returns a string containing any bytes recognized as a TOML string.

TOML strings are everything found between two double or single quotation marks ("/').

#fn (&Scanner) extract_multiline_string

direct_array_accessinline
fn (mut s &Scanner) extract_multiline_string() !string

extract_multiline_string collects and returns a string containing any bytes recognized as a TOML string.

TOML strings are everything found between two double or single quotation marks ("/').

#fn (&Scanner) handle_escapes

fn (mut s &Scanner) handle_escapes(quote u8, is_multiline bool) (string, int)

handle_escapes returns any escape character sequence.

For escape sequence validation see Checker.check_quoted_escapes.

#fn (&Scanner) extract_number

direct_array_accessinline
fn (mut s &Scanner) extract_number() !string

extract_number collects and returns a string containing any bytes recognized as a TOML number except for "(+/-)nan" and "(+/-)inf".

TOML numbers can include digits 0-9 and _.

#fn (&Scanner) extract_nan_or_inf_number

direct_array_accessinline
fn (mut s &Scanner) extract_nan_or_inf_number() !string

extract_nan_or_inf_number collects and returns a string containing any bytes recognized as infinity or not-a-number TOML numbers.

#fn (Scanner) excerpt

fn (s Scanner) excerpt(pos int, margin int) string

excerpt returns a string excerpt of the input text centered at pos. The margin argument defines how many chacters on each side of pos is returned

#fn (Scanner) state

fn (s Scanner) state() State

state returns a read-only view of the scanner's internal state.

#fn (&Scanner) validate_and_skip_headers

fn (mut s &Scanner) validate_and_skip_headers() !

#fn (&Scanner) check_utf16_or_32_bom

fn (mut s &Scanner) check_utf16_or_32_bom() !

#struct State

pub struct State {
pub:
	col     int // current column number (x coordinate)
	line_nr int = 1 // current line number (y coordinate)
	pos     int // current flat/index position in the `text` field
}

State is a read-only copy of the scanner's internal state.

See also Scanner.state().

#struct Config

pub struct Config {
pub:
	input               input.Config
	tokenize_formatting bool = true // if true, generate tokens for `\n`, ` `, `\t`, `\r` etc.
}

Config is used to configure a Scanner instance.

Only one of the fields text and file_path is allowed to be set at time of configuration.

Interfaces

This section is empty.

Enums

This section is empty.