pub fn highlight_rust_lexical(text: &str) -> Vec<SyntaxToken>Expand description
Tokenizes a useful ASCII-oriented subset of Rust without parsing.
The deterministic, linear scan recognizes line comments, quoted strings and
chars, raw strings, lifetimes, numeric/identifier runs, keywords, delimiters,
and operators. It then marks an identifier following fn and identifiers
followed by optional ASCII whitespace plus ! as functions. Non-ASCII
identifier bytes and unrecognized punctuation are skipped. Unterminated
strings/comments extend to EOF; returned ranges are ordered UTF-8 bytes.
ยงExamples
use ailloli_ui_editor::code::{highlight_rust_lexical, SyntaxKind};
let text = "fn main() { println!(\"hi\"); }";
let tokens = highlight_rust_lexical(text);
assert!(tokens.iter().any(|t| t.kind == SyntaxKind::Keyword && &text[t.range.clone()] == "fn"));
assert!(tokens.iter().any(|t| t.kind == SyntaxKind::Function && &text[t.range.clone()] == "main"));
assert!(tokens.iter().any(|t| t.kind == SyntaxKind::String && &text[t.range.clone()] == "\"hi\""));