Enum rustc_lexer::TokenKind
source · pub enum TokenKind {
Show 39 variants
LineComment {
doc_style: Option<DocStyle>,
},
BlockComment {
doc_style: Option<DocStyle>,
terminated: bool,
},
Whitespace,
Ident,
InvalidIdent,
RawIdent,
UnknownPrefix,
InvalidPrefix,
Literal {
kind: LiteralKind,
suffix_start: u32,
},
Lifetime {
starts_with_number: bool,
},
Semi,
Comma,
Dot,
OpenParen,
CloseParen,
OpenBrace,
CloseBrace,
OpenBracket,
CloseBracket,
At,
Pound,
Tilde,
Question,
Colon,
Dollar,
Eq,
Bang,
Lt,
Gt,
Minus,
And,
Or,
Plus,
Star,
Slash,
Caret,
Percent,
Unknown,
Eof,
}
Expand description
Enum representing common lexeme types.
Variants§
LineComment
“// comment”
BlockComment
/* block comment */
Block comments can be recursive, so a sequence like /* /* */
will not be considered terminated and will result in a parsing error.
Whitespace
Any whitespace character sequence.
Ident
“ident” or “continue”
At this step, keywords are also considered identifiers.
InvalidIdent
Like the above, but containing invalid unicode codepoints.
RawIdent
“r#ident”
UnknownPrefix
An unknown prefix, like foo#
, foo'
, foo"
.
Note that only the
prefix (foo
) is included in the token, not the separator (which is
lexed as its own distinct token). In Rust 2021 and later, reserved
prefixes are reported as errors; in earlier editions, they result in a
(allowed by default) lint, and are treated as regular identifier
tokens.
InvalidPrefix
Similar to the above, but always an error on every edition. This is used for emoji identifier recovery, as those are not meant to be ever accepted.
Literal
Examples: 12u8
, 1.0e-40
, b"123"
. Note that _
is an invalid
suffix, but may be present here on string and float literals. Users of
this type will need to check for and reject that case.
See LiteralKind for more details.
Lifetime
“’a”
Semi
“;”
Comma
“,”
Dot
“.”
OpenParen
“(”
CloseParen
“)”
OpenBrace
“{”
CloseBrace
“}”
OpenBracket
“[”
CloseBracket
“]”
At
“@”
Pound
“#”
Tilde
“~”
Question
“?”
Colon
“:”
Dollar
“$”
Eq
“=”
Bang
“!”
Lt
“<”
Gt
“>”
Minus
“-”
And
“&”
Or
“|”
Plus
“+”
Star
“*”
Slash
“/”
Caret
“^”
Percent
“%”
Unknown
Unknown token, not expected by the lexer, e.g. “№”
Eof
End of input.
Trait Implementations§
impl Copy for TokenKind
impl Eq for TokenKind
impl StructuralPartialEq for TokenKind
Auto Trait Implementations§
impl Freeze for TokenKind
impl RefUnwindSafe for TokenKind
impl Send for TokenKind
impl Sync for TokenKind
impl Unpin for TokenKind
impl UnwindSafe for TokenKind
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
source§unsafe fn clone_to_uninit(&self, dst: *mut T)
unsafe fn clone_to_uninit(&self, dst: *mut T)
clone_to_uninit
)Layout§
Note: Most layout information is completely unstable and may even differ between compilations. The only exception is types with certain repr(...)
attributes. Please see the Rust Reference's “Type Layout” chapter for details on type layout guarantees.
Size: 8 bytes
Size for each variant:
LineComment
: 1 byteBlockComment
: 2 bytesWhitespace
: 0 bytesIdent
: 0 bytesInvalidIdent
: 0 bytesRawIdent
: 0 bytesUnknownPrefix
: 0 bytesInvalidPrefix
: 0 bytesLiteral
: 7 bytesLifetime
: 1 byteSemi
: 0 bytesComma
: 0 bytesDot
: 0 bytesOpenParen
: 0 bytesCloseParen
: 0 bytesOpenBrace
: 0 bytesCloseBrace
: 0 bytesOpenBracket
: 0 bytesCloseBracket
: 0 bytesAt
: 0 bytesPound
: 0 bytesTilde
: 0 bytesQuestion
: 0 bytesColon
: 0 bytesDollar
: 0 bytesEq
: 0 bytesBang
: 0 bytesLt
: 0 bytesGt
: 0 bytesMinus
: 0 bytesAnd
: 0 bytesOr
: 0 bytesPlus
: 0 bytesStar
: 0 bytesSlash
: 0 bytesCaret
: 0 bytesPercent
: 0 bytesUnknown
: 0 bytesEof
: 0 bytes