Enum rustc_lexer::TokenKind
source · [−]pub enum TokenKind {
Show 37 variants
LineComment {
doc_style: Option<DocStyle>,
},
BlockComment {
doc_style: Option<DocStyle>,
terminated: bool,
},
Whitespace,
Ident,
InvalidIdent,
RawIdent,
UnknownPrefix,
Literal {
kind: LiteralKind,
suffix_start: usize,
},
Lifetime {
starts_with_number: bool,
},
Semi,
Comma,
Dot,
OpenParen,
CloseParen,
OpenBrace,
CloseBrace,
OpenBracket,
CloseBracket,
At,
Pound,
Tilde,
Question,
Colon,
Dollar,
Eq,
Bang,
Lt,
Gt,
Minus,
And,
Or,
Plus,
Star,
Slash,
Caret,
Percent,
Unknown,
}
Expand description
Enum representing common lexeme types.
Variants
LineComment
“// comment”
BlockComment
/* block comment */
Block comments can be recursive, so the sequence like /* /* */
will not be considered terminated and will result in a parsing error.
Whitespace
Any whitespace characters sequence.
Ident
“ident” or “continue” At this step keywords are also considered identifiers.
InvalidIdent
Like the above, but containing invalid unicode codepoints.
RawIdent
“r#ident”
UnknownPrefix
An unknown prefix like foo#
, foo'
, foo"
. Note that only the
prefix (foo
) is included in the token, not the separator (which is
lexed as its own distinct token). In Rust 2021 and later, reserved
prefixes are reported as errors; in earlier editions, they result in a
(allowed by default) lint, and are treated as regular identifier
tokens.
Literal
“12_u8”, “1.0e-40”, “b“123”“. See LiteralKind
for more details.
Lifetime
Fields
starts_with_number: bool
“’a”
Semi
“;”
Comma
“,”
Dot
“.”
OpenParen
“(”
CloseParen
“)”
OpenBrace
“{”
CloseBrace
“}”
OpenBracket
“[”
CloseBracket
“]”
At
“@”
Pound
“#”
Tilde
“~”
Question
“?”
Colon
“:”
Dollar
“$”
Eq
“=”
Bang
“!”
Lt
“<”
Gt
“>”
Minus
“-”
And
“&”
Or
“|”
Plus
“+”
Star
“*”
Slash
“/”
Caret
“^”
Percent
“%”
Unknown
Unknown token, not expected by the lexer, e.g. “№”
Trait Implementations
sourceimpl Ord for TokenKind
impl Ord for TokenKind
sourceimpl PartialOrd<TokenKind> for TokenKind
impl PartialOrd<TokenKind> for TokenKind
sourcefn partial_cmp(&self, other: &TokenKind) -> Option<Ordering>
fn partial_cmp(&self, other: &TokenKind) -> Option<Ordering>
This method returns an ordering between self
and other
values if one exists. Read more
1.0.0 · sourcefn lt(&self, other: &Rhs) -> bool
fn lt(&self, other: &Rhs) -> bool
This method tests less than (for self
and other
) and is used by the <
operator. Read more
1.0.0 · sourcefn le(&self, other: &Rhs) -> bool
fn le(&self, other: &Rhs) -> bool
This method tests less than or equal to (for self
and other
) and is used by the <=
operator. Read more
impl Copy for TokenKind
impl Eq for TokenKind
impl StructuralEq for TokenKind
impl StructuralPartialEq for TokenKind
Auto Trait Implementations
impl RefUnwindSafe for TokenKind
impl Send for TokenKind
impl Sync for TokenKind
impl Unpin for TokenKind
impl UnwindSafe for TokenKind
Blanket Implementations
sourceimpl<T> BorrowMut<T> for T where
T: ?Sized,
impl<T> BorrowMut<T> for T where
T: ?Sized,
const: unstable · sourcefn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Layout
Note: Most layout information is completely unstable and may even differ between compilations. The only exception is types with certain repr(...)
attributes. Please see the Rust Reference’s “Type Layout” chapter for details on type layout guarantees.
Size: 64 bytes
Size for each variant:
LineComment
: 1 byteBlockComment
: 2 bytesWhitespace
: 0 bytesIdent
: 0 bytesInvalidIdent
: 0 bytesRawIdent
: 0 bytesUnknownPrefix
: 0 bytesLiteral
: 63 bytesLifetime
: 1 byteSemi
: 0 bytesComma
: 0 bytesDot
: 0 bytesOpenParen
: 0 bytesCloseParen
: 0 bytesOpenBrace
: 0 bytesCloseBrace
: 0 bytesOpenBracket
: 0 bytesCloseBracket
: 0 bytesAt
: 0 bytesPound
: 0 bytesTilde
: 0 bytesQuestion
: 0 bytesColon
: 0 bytesDollar
: 0 bytesEq
: 0 bytesBang
: 0 bytesLt
: 0 bytesGt
: 0 bytesMinus
: 0 bytesAnd
: 0 bytesOr
: 0 bytesPlus
: 0 bytesStar
: 0 bytesSlash
: 0 bytesCaret
: 0 bytesPercent
: 0 bytesUnknown
: 0 bytes