Module twig::tokens

Produces a token stream from source template.

Summary

This module is capable of taking a Twig input template, for example, this one:

Hello
{% if world %}
    world
{% else %}
    {{ other }}
{% endif %}

And chopping it into tokens like these:

Ok(TokenRef { value: Text("Hello\n"), line: 1 })
Ok(TokenRef { value: BlockStart, line: 2 })
Ok(TokenRef { value: Name("if"), line: 2 })
Ok(TokenRef { value: Name("world"), line: 2 })
Ok(TokenRef { value: BlockEnd, line: 2 })
Ok(TokenRef { value: Text("    world\n"), line: 3 })
Ok(TokenRef { value: BlockStart, line: 4 })
Ok(TokenRef { value: Name("else"), line: 4 })
Ok(TokenRef { value: BlockEnd, line: 4 })
Ok(TokenRef { value: Text("    "), line: 5 })
Ok(TokenRef { value: VarStart, line: 5 })
Ok(TokenRef { value: Name("other"), line: 5 })
Ok(TokenRef { value: VarEnd, line: 5 })
Ok(TokenRef { value: Text("\n"), line: 5 })
Ok(TokenRef { value: BlockStart, line: 6 })
Ok(TokenRef { value: Name("endif"), line: 6 })
Ok(TokenRef { value: BlockEnd, line: 6 })

Example code for this:

use twig::environment::Environment;
use twig::tokens::Lexer;

let env = Environment::default().init_all();
let lexer = Lexer::default(&env.lexing);

for token in lexer.tokens(source) {
    println!("{:?}", token);
}

Structs

Lexer	Parses template file and converts it to a stream of tokens.
LexerOptions	Lexer options.
TokenIter
TokenRef	Lexer output token, lexer's output and parser's input.

Enums

Const
ConstNumber	Parsed twig number representation.
ConstNumberRef	Parsed twig number representation.
ConstRef
TokenValue	Token value.
TokenValueRef	Token value.