pub enum TokenKind {
Show 37 variants LineComment { doc_style: Option<DocStyle>, }, BlockComment { doc_style: Option<DocStyle>, terminated: bool, }, Whitespace, Ident, InvalidIdent, RawIdent, UnknownPrefix, Literal { kind: LiteralKind, suffix_start: usize, }, Lifetime { starts_with_number: bool, }, Semi, Comma, Dot, OpenParen, CloseParen, OpenBrace, CloseBrace, OpenBracket, CloseBracket, At, Pound, Tilde, Question, Colon, Dollar, Eq, Bang, Lt, Gt, Minus, And, Or, Plus, Star, Slash, Caret, Percent, Unknown,
}
Expand description

Enum representing common lexeme types.

Variants

LineComment

Fields

doc_style: Option<DocStyle>

“// comment”

BlockComment

Fields

doc_style: Option<DocStyle>
terminated: bool

/* block comment */

Block comments can be recursive, so the sequence like /* /* */ will not be considered terminated and will result in a parsing error.

Whitespace

Any whitespace characters sequence.

Ident

“ident” or “continue” At this step keywords are also considered identifiers.

InvalidIdent

Like the above, but containing invalid unicode codepoints.

RawIdent

“r#ident”

UnknownPrefix

An unknown prefix like foo#, foo', foo". Note that only the prefix (foo) is included in the token, not the separator (which is lexed as its own distinct token). In Rust 2021 and later, reserved prefixes are reported as errors; in earlier editions, they result in a (allowed by default) lint, and are treated as regular identifier tokens.

Literal

Fields

suffix_start: usize

“12_u8”, “1.0e-40”, “b“123”“. See LiteralKind for more details.

Lifetime

Fields

starts_with_number: bool

“’a”

Semi

“;”

Comma

“,”

Dot

“.”

OpenParen

“(”

CloseParen

“)”

OpenBrace

“{”

CloseBrace

“}”

OpenBracket

“[”

CloseBracket

“]”

At

“@”

Pound

“#”

Tilde

“~”

Question

“?”

Colon

“:”

Dollar

“$”

Eq

“=”

Bang

“!”

Lt

“<”

Gt

“>”

Minus

“-”

And

“&”

Or

“|”

Plus

“+”

Star

“*”

Slash

“/”

Caret

“^”

Percent

“%”

Unknown

Unknown token, not expected by the lexer, e.g. “№”

Trait Implementations

Returns a copy of the value. Read more

Performs copy-assignment from source. Read more

Formats the value using the given formatter. Read more

This method returns an Ordering between self and other. Read more

Compares and returns the maximum of two values. Read more

Compares and returns the minimum of two values. Read more

Restrict a value to a certain interval. Read more

This method tests for self and other values to be equal, and is used by ==. Read more

This method tests for !=.

This method returns an ordering between self and other values if one exists. Read more

This method tests less than (for self and other) and is used by the < operator. Read more

This method tests less than or equal to (for self and other) and is used by the <= operator. Read more

This method tests greater than (for self and other) and is used by the > operator. Read more

This method tests greater than or equal to (for self and other) and is used by the >= operator. Read more

Auto Trait Implementations

Blanket Implementations

Gets the TypeId of self. Read more

Immutably borrows from an owned value. Read more

Mutably borrows from an owned value. Read more

Returns the argument unchanged.

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

The resulting type after obtaining ownership.

Creates owned data from borrowed data, usually by cloning. Read more

Uses borrowed data to replace owned data, usually by cloning. Read more

The type returned in the event of a conversion error.

Performs the conversion.

The type returned in the event of a conversion error.

Performs the conversion.

Layout

Note: Most layout information is completely unstable and may even differ between compilations. The only exception is types with certain repr(...) attributes. Please see the Rust Reference’s “Type Layout” chapter for details on type layout guarantees.

Size: 64 bytes

Size for each variant:

  • LineComment: 1 byte
  • BlockComment: 2 bytes
  • Whitespace: 0 bytes
  • Ident: 0 bytes
  • InvalidIdent: 0 bytes
  • RawIdent: 0 bytes
  • UnknownPrefix: 0 bytes
  • Literal: 63 bytes
  • Lifetime: 1 byte
  • Semi: 0 bytes
  • Comma: 0 bytes
  • Dot: 0 bytes
  • OpenParen: 0 bytes
  • CloseParen: 0 bytes
  • OpenBrace: 0 bytes
  • CloseBrace: 0 bytes
  • OpenBracket: 0 bytes
  • CloseBracket: 0 bytes
  • At: 0 bytes
  • Pound: 0 bytes
  • Tilde: 0 bytes
  • Question: 0 bytes
  • Colon: 0 bytes
  • Dollar: 0 bytes
  • Eq: 0 bytes
  • Bang: 0 bytes
  • Lt: 0 bytes
  • Gt: 0 bytes
  • Minus: 0 bytes
  • And: 0 bytes
  • Or: 0 bytes
  • Plus: 0 bytes
  • Star: 0 bytes
  • Slash: 0 bytes
  • Caret: 0 bytes
  • Percent: 0 bytes
  • Unknown: 0 bytes