Expand description
§Token Streams
TokenStream
s represent syntactic objects before they are converted into ASTs.
A TokenStream
is, roughly speaking, a sequence of TokenTree
s,
which are themselves a single Token
or a Delimited
subsequence of tokens.
§Ownership
TokenStream
s are persistent data structures constructed as ropes with reference
counted-children. In general, this means that calling an operation on a TokenStream
(such as slice
) produces an entirely new TokenStream
from the borrowed reference to
the original. This essentially coerces TokenStream
s into “views” of their subparts,
and a borrowed TokenStream
is sufficient to build an owned TokenStream
without taking
ownership of the original.
Modules§
Structs§
- Attr
Token Stream - An
AttrTokenStream
is similar to aTokenStream
, but with extra information about the tokens for attribute targets. This is used during expansion to perform early cfg-expansion, and to process attributes during proc-macro invocations. - Attrs
Target - Stores the tokens for an attribute target, along with its attributes.
- Delim
Spacing - Delim
Span - Lazy
Attr Token Stream - A lazy version of
AttrTokenStream
, which defers creation of an actualAttrTokenStream
until it is needed. - Node
Range - A token range within an individual AST node’s (lazy) token stream, i.e.
relative to that node’s first token. Distinct from
ParserRange
so the two kinds of range can’t be mixed up. - Parser
Range - A token range within a
Parser
’s full token stream. - Token
Cursor - A
TokenStream
cursor that producesToken
s. It’s a bit odd that we (a) lex tokens into a nice tree structure (TokenStream
), and then (b) use this type to emit them as a linear sequence. But a linear sequence is what the parser expects, for the most part. - Token
Stream - A
TokenStream
is an abstract sequence of tokens, organized intoTokenTree
s. - Token
Stream Iter - Token
Tree Cursor
Enums§
- Attr
Token Tree - Like
TokenTree
, but forAttrTokenStream
. - Flat
Token 🔒 - A helper struct used when building an
AttrTokenStream
from aLazyAttrTokenStream
. Both delimiter and non-delimited tokens are stored asFlatToken::Token
. A vector ofFlatToken
s is then ‘parsed’ to build up anAttrTokenStream
with nestedAttrTokenTree::Delimited
tokens. - Lazy
Attr 🔒Token Stream Inner - Spacing
- Indicates whether a token can join with the following token to form a
compound token. Used for conversions to
proc_macro::Spacing
. Also used to guide pretty-printing, which is where theJointHidden
value (which isn’t part ofproc_macro::Spacing
) comes in useful. - Token
Tree - Part of a
TokenStream
.
Functions§
- _dummy 🔒
- attrs_
and_ 🔒tokens_ to_ token_ trees - make_
attr_ 🔒token_ stream - Converts a flattened iterator of tokens (including open and close delimiter tokens) into an
AttrTokenStream
, creating anAttrTokenTree::Delimited
for each matching pair of open and close delims.
Type Aliases§
- Node
Replacement - See the comment on
ParserReplacement
. - Parser
Replacement - Indicates a range of tokens that should be replaced by an
AttrsTarget
(replacement) or be replaced by nothing (deletion). This is used in two places during token collection.