Enum rustc_lexer::TokenKind

source ·
pub enum TokenKind {
Show 39 variants LineComment { doc_style: Option<DocStyle>, }, BlockComment { doc_style: Option<DocStyle>, terminated: bool, }, Whitespace, Ident, InvalidIdent, RawIdent, UnknownPrefix, InvalidPrefix, Literal { kind: LiteralKind, suffix_start: u32, }, Lifetime { starts_with_number: bool, }, Semi, Comma, Dot, OpenParen, CloseParen, OpenBrace, CloseBrace, OpenBracket, CloseBracket, At, Pound, Tilde, Question, Colon, Dollar, Eq, Bang, Lt, Gt, Minus, And, Or, Plus, Star, Slash, Caret, Percent, Unknown, Eof,
}
Expand description

Enum representing common lexeme types.

Variants§

§

LineComment

“// comment”

Fields

§doc_style: Option<DocStyle>
§

BlockComment

/* block comment */

Block comments can be recursive, so a sequence like /* /* */ will not be considered terminated and will result in a parsing error.

Fields

§doc_style: Option<DocStyle>
§terminated: bool
§

Whitespace

Any whitespace character sequence.

§

Ident

“ident” or “continue”

At this step, keywords are also considered identifiers.

§

InvalidIdent

Like the above, but containing invalid unicode codepoints.

§

RawIdent

“r#ident”

§

UnknownPrefix

An unknown prefix, like foo#, foo', foo".

Note that only the prefix (foo) is included in the token, not the separator (which is lexed as its own distinct token). In Rust 2021 and later, reserved prefixes are reported as errors; in earlier editions, they result in a (allowed by default) lint, and are treated as regular identifier tokens.

§

InvalidPrefix

Similar to the above, but always an error on every edition. This is used for emoji identifier recovery, as those are not meant to be ever accepted.

§

Literal

Examples: 12u8, 1.0e-40, b"123". Note that _ is an invalid suffix, but may be present here on string and float literals. Users of this type will need to check for and reject that case.

See LiteralKind for more details.

Fields

§suffix_start: u32
§

Lifetime

“’a”

Fields

§starts_with_number: bool
§

Semi

“;”

§

Comma

“,”

§

Dot

“.”

§

OpenParen

“(”

§

CloseParen

“)”

§

OpenBrace

“{”

§

CloseBrace

“}”

§

OpenBracket

“[”

§

CloseBracket

“]”

§

At

“@”

§

Pound

“#”

§

Tilde

“~”

§

Question

“?”

§

Colon

“:”

§

Dollar

“$”

§

Eq

“=”

§

Bang

“!”

§

Lt

“<”

§

Gt

“>”

§

Minus

“-”

§

And

“&”

§

Or

“|”

§

Plus

“+”

§

Star

“*”

§

Slash

“/”

§

Caret

“^”

§

Percent

“%”

§

Unknown

Unknown token, not expected by the lexer, e.g. “№”

§

Eof

End of input.

Trait Implementations§

source§

impl Clone for TokenKind

source§

fn clone(&self) -> TokenKind

Returns a copy of the value. Read more
1.0.0 · source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more
source§

impl Debug for TokenKind

source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
source§

impl PartialEq for TokenKind

source§

fn eq(&self, other: &TokenKind) -> bool

Tests for self and other values to be equal, and is used by ==.
1.0.0 · source§

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.
source§

impl Copy for TokenKind

source§

impl Eq for TokenKind

source§

impl StructuralPartialEq for TokenKind

Auto Trait Implementations§

Blanket Implementations§

source§

impl<T> Any for T
where T: 'static + ?Sized,

source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
source§

impl<T> Borrow<T> for T
where T: ?Sized,

source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
source§

impl<T> CloneToUninit for T
where T: Clone,

source§

unsafe fn clone_to_uninit(&self, dst: *mut T)

🔬This is a nightly-only experimental API. (clone_to_uninit)
Performs copy-assignment from self to dst. Read more
source§

impl<T> From<T> for T

source§

fn from(t: T) -> T

Returns the argument unchanged.

source§

impl<T, U> Into<U> for T
where U: From<T>,

source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

source§

impl<T> ToOwned for T
where T: Clone,

source§

type Owned = T

The resulting type after obtaining ownership.
source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

source§

type Error = Infallible

The type returned in the event of a conversion error.
source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.

Layout§

Note: Most layout information is completely unstable and may even differ between compilations. The only exception is types with certain repr(...) attributes. Please see the Rust Reference's “Type Layout” chapter for details on type layout guarantees.

Size: 8 bytes

Size for each variant:

  • LineComment: 1 byte
  • BlockComment: 2 bytes
  • Whitespace: 0 bytes
  • Ident: 0 bytes
  • InvalidIdent: 0 bytes
  • RawIdent: 0 bytes
  • UnknownPrefix: 0 bytes
  • InvalidPrefix: 0 bytes
  • Literal: 7 bytes
  • Lifetime: 1 byte
  • Semi: 0 bytes
  • Comma: 0 bytes
  • Dot: 0 bytes
  • OpenParen: 0 bytes
  • CloseParen: 0 bytes
  • OpenBrace: 0 bytes
  • CloseBrace: 0 bytes
  • OpenBracket: 0 bytes
  • CloseBracket: 0 bytes
  • At: 0 bytes
  • Pound: 0 bytes
  • Tilde: 0 bytes
  • Question: 0 bytes
  • Colon: 0 bytes
  • Dollar: 0 bytes
  • Eq: 0 bytes
  • Bang: 0 bytes
  • Lt: 0 bytes
  • Gt: 0 bytes
  • Minus: 0 bytes
  • And: 0 bytes
  • Or: 0 bytes
  • Plus: 0 bytes
  • Star: 0 bytes
  • Slash: 0 bytes
  • Caret: 0 bytes
  • Percent: 0 bytes
  • Unknown: 0 bytes
  • Eof: 0 bytes