[][src]Function rustc_lexer::tokenize

pub fn tokenize(input: &str) -> impl Iterator<Item = Token> + '_
⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

Creates an iterator that produces tokens from the input string.