Module tokenstream

Source
Expand description

§Token Streams

TokenStreams represent syntactic objects before they are converted into ASTs. A TokenStream is, roughly speaking, a sequence of TokenTrees, which are themselves a single Token or a Delimited subsequence of tokens.

§Ownership

TokenStreams are persistent data structures constructed as ropes with reference counted-children. In general, this means that calling an operation on a TokenStream (such as slice) produces an entirely new TokenStream from the borrowed reference to the original. This essentially coerces TokenStreams into “views” of their subparts, and a borrowed TokenStream is sufficient to build an owned TokenStream without taking ownership of the original.

Modules§

size_asserts 🔒

Structs§

AttrTokenStream
An AttrTokenStream is similar to a TokenStream, but with extra information about the tokens for attribute targets. This is used during expansion to perform early cfg-expansion, and to process attributes during proc-macro invocations.
AttrsTarget
Stores the tokens for an attribute target, along with its attributes.
DelimSpacing
DelimSpan
LazyAttrTokenStream
A lazy version of AttrTokenStream, which defers creation of an actual AttrTokenStream until it is needed.
NodeRange
A token range within an individual AST node’s (lazy) token stream, i.e. relative to that node’s first token. Distinct from ParserRange so the two kinds of range can’t be mixed up.
ParserRange
A token range within a Parser’s full token stream.
TokenCursor
A TokenStream cursor that produces Tokens. It’s a bit odd that we (a) lex tokens into a nice tree structure (TokenStream), and then (b) use this type to emit them as a linear sequence. But a linear sequence is what the parser expects, for the most part.
TokenStream
A TokenStream is an abstract sequence of tokens, organized into TokenTrees.
TokenStreamIter
TokenTreeCursor

Enums§

AttrTokenTree
Like TokenTree, but for AttrTokenStream.
FlatToken 🔒
A helper struct used when building an AttrTokenStream from a LazyAttrTokenStream. Both delimiter and non-delimited tokens are stored as FlatToken::Token. A vector of FlatTokens is then ‘parsed’ to build up an AttrTokenStream with nested AttrTokenTree::Delimited tokens.
LazyAttrTokenStreamInner 🔒
Spacing
Indicates whether a token can join with the following token to form a compound token. Used for conversions to proc_macro::Spacing. Also used to guide pretty-printing, which is where the JointHidden value (which isn’t part of proc_macro::Spacing) comes in useful.
TokenTree
Part of a TokenStream.

Functions§

_dummy 🔒
attrs_and_tokens_to_token_trees 🔒
make_attr_token_stream 🔒
Converts a flattened iterator of tokens (including open and close delimiter tokens) into an AttrTokenStream, creating an AttrTokenTree::Delimited for each matching pair of open and close delims.

Type Aliases§

NodeReplacement
See the comment on ParserReplacement.
ParserReplacement
Indicates a range of tokens that should be replaced by an AttrsTarget (replacement) or be replaced by nothing (deletion). This is used in two places during token collection.