pub enum TokenKind {
Show 39 variants
Eq,
Lt,
Le,
EqEq,
Ne,
Ge,
Gt,
AndAnd,
OrOr,
Not,
Tilde,
BinOp(BinOpToken),
BinOpEq(BinOpToken),
At,
Dot,
DotDot,
DotDotDot,
DotDotEq,
Comma,
Semi,
Colon,
PathSep,
RArrow,
LArrow,
FatArrow,
Pound,
Dollar,
Question,
SingleQuote,
OpenDelim(Delimiter),
CloseDelim(Delimiter),
Literal(Lit),
Ident(Symbol, IdentIsRaw),
NtIdent(Ident, IdentIsRaw),
Lifetime(Symbol, IdentIsRaw),
NtLifetime(Ident, IdentIsRaw),
Interpolated(Lrc<Nonterminal>),
DocComment(CommentKind, AttrStyle, Symbol),
Eof,
}
Variants§
Eq
=
Lt
<
Le
<=
EqEq
==
Ne
!=
Ge
>=
Gt
>
AndAnd
&&
OrOr
||
Not
!
Tilde
~
BinOp(BinOpToken)
BinOpEq(BinOpToken)
At
@
Dot
.
DotDot
..
DotDotDot
...
DotDotEq
..=
Comma
,
Semi
;
Colon
:
PathSep
::
RArrow
->
LArrow
<-
FatArrow
=>
Pound
#
Dollar
$
Question
?
SingleQuote
Used by proc macros for representing lifetimes, not generated by lexer right now.
OpenDelim(Delimiter)
An opening delimiter (e.g., {
).
CloseDelim(Delimiter)
A closing delimiter (e.g., }
).
Literal(Lit)
Ident(Symbol, IdentIsRaw)
Identifier token.
Do not forget about NtIdent
when you want to match on identifiers.
It’s recommended to use Token::(ident,uninterpolate,uninterpolated_span)
to
treat regular and interpolated identifiers in the same way.
NtIdent(Ident, IdentIsRaw)
This identifier (and its span) is the identifier passed to the
declarative macro. The span in the surrounding Token
is the span of
the ident
metavariable in the macro’s RHS.
Lifetime(Symbol, IdentIsRaw)
Lifetime identifier token.
Do not forget about NtLifetime
when you want to match on lifetime identifiers.
It’s recommended to use Token::(lifetime,uninterpolate,uninterpolated_span)
to
treat regular and interpolated lifetime identifiers in the same way.
NtLifetime(Ident, IdentIsRaw)
This identifier (and its span) is the lifetime passed to the
declarative macro. The span in the surrounding Token
is the span of
the lifetime
metavariable in the macro’s RHS.
Interpolated(Lrc<Nonterminal>)
An embedded AST node, as produced by a macro. This only exists for historical reasons. We’d like to get rid of it, for multiple reasons.
- It’s conceptually very strange. Saying a token can contain an AST node is like saying, in natural language, that a word can contain a sentence.
- It requires special handling in a bunch of places in the parser.
- It prevents
Token
from implementingCopy
. It adds complexity and likely slows things down. Please don’t add new occurrences of this token kind!
The span in the surrounding Token
is that of the metavariable in the
macro’s RHS. The span within the Nonterminal is that of the fragment
passed to the macro at the call site.
DocComment(CommentKind, AttrStyle, Symbol)
A doc comment token.
Symbol
is the doc comment’s data excluding its “quotes” (///
, /**
, etc)
similarly to symbols in string literal tokens.
Eof
End Of File
Implementations§
source§impl TokenKind
impl TokenKind
pub fn lit(kind: LitKind, symbol: Symbol, suffix: Option<Symbol>) -> TokenKind
sourcepub fn break_two_token_op(&self, n: u32) -> Option<(TokenKind, TokenKind)>
pub fn break_two_token_op(&self, n: u32) -> Option<(TokenKind, TokenKind)>
An approximation to proc-macro-style single-character operators used by
rustc parser. If the operator token can be broken into two tokens, the
first of which has n
(1 or 2) chars, then this function performs that
operation, otherwise it returns None
.
sourcepub fn similar_tokens(&self) -> Option<Vec<TokenKind>>
pub fn similar_tokens(&self) -> Option<Vec<TokenKind>>
Returns tokens that are likely to be typed accidentally instead of the current token. Enables better error recovery when the wrong token is found.
pub fn should_end_const_arg(&self) -> bool
Trait Implementations§
source§impl<__D: SpanDecoder> Decodable<__D> for TokenKind
impl<__D: SpanDecoder> Decodable<__D> for TokenKind
source§impl<__E: SpanEncoder> Encodable<__E> for TokenKind
impl<__E: SpanEncoder> Encodable<__E> for TokenKind
source§impl<__CTX> HashStable<__CTX> for TokenKindwhere
__CTX: HashStableContext,
impl<__CTX> HashStable<__CTX> for TokenKindwhere
__CTX: HashStableContext,
fn hash_stable(&self, __hcx: &mut __CTX, __hasher: &mut StableHasher)
impl StructuralPartialEq for TokenKind
Auto Trait Implementations§
impl Freeze for TokenKind
impl !RefUnwindSafe for TokenKind
impl !Send for TokenKind
impl !Sync for TokenKind
impl Unpin for TokenKind
impl !UnwindSafe for TokenKind
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
source§unsafe fn clone_to_uninit(&self, dst: *mut T)
unsafe fn clone_to_uninit(&self, dst: *mut T)
clone_to_uninit
)source§impl<T> Instrument for T
impl<T> Instrument for T
source§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
source§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
source§impl<T> IntoEither for T
impl<T> IntoEither for T
source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self
into a Left
variant of Either<Self, Self>
if into_left
is true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read moresource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self
into a Left
variant of Either<Self, Self>
if into_left(&self)
returns true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read moresource§impl<T> WithSubscriber for T
impl<T> WithSubscriber for T
source§fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self>
fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self>
source§fn with_current_subscriber(self) -> WithDispatch<Self>
fn with_current_subscriber(self) -> WithDispatch<Self>
impl<'a, T> Captures<'a> for Twhere
T: ?Sized,
Layout§
Note: Most layout information is completely unstable and may even differ between compilations. The only exception is types with certain repr(...)
attributes. Please see the Rust Reference's “Type Layout” chapter for details on type layout guarantees.
Size: 16 bytes
Size for each variant:
Eq
: 0 bytesLt
: 0 bytesLe
: 0 bytesEqEq
: 0 bytesNe
: 0 bytesGe
: 0 bytesGt
: 0 bytesAndAnd
: 0 bytesOrOr
: 0 bytesNot
: 0 bytesTilde
: 0 bytesBinOp
: 1 byteBinOpEq
: 1 byteAt
: 0 bytesDot
: 0 bytesDotDot
: 0 bytesDotDotDot
: 0 bytesDotDotEq
: 0 bytesComma
: 0 bytesSemi
: 0 bytesColon
: 0 bytesPathSep
: 0 bytesRArrow
: 0 bytesLArrow
: 0 bytesFatArrow
: 0 bytesPound
: 0 bytesDollar
: 0 bytesQuestion
: 0 bytesSingleQuote
: 0 bytesOpenDelim
: 1 byteCloseDelim
: 1 byteLiteral
: 15 bytesIdent
: 7 bytesNtIdent
: 15 bytesLifetime
: 7 bytesNtLifetime
: 15 bytesInterpolated
: 15 bytesDocComment
: 7 bytesEof
: 0 bytes