Module tokenstream

Token Streams

TokenStreams represent syntactic objects before they are converted into ASTs. A TokenStream is, roughly speaking, a sequence of TokenTrees, which are themselves a single Token or a Delimited subsequence of tokens.

Ownership

TokenStreams are persistent data structures constructed as ropes with reference counted-children. In general, this means that calling an operation on a TokenStream (such as slice) produces an entirely new TokenStream from the borrowed reference to the original. This essentially coerces TokenStreams into “views” of their subparts, and a borrowed TokenStream is sufficient to build an owned TokenStream without taking ownership of the original.

Modules

size_asserts 🔒

Structs

AttrTokenStream

An AttrTokenStream is similar to a TokenStream, but with extra information about the tokens for attribute targets. This is used during expansion to perform early cfg-expansion, and to process attributes during proc-macro invocations.

AttrsTarget

Stores the tokens for an attribute target, along with its attributes.

DelimSpacing

DelimSpan

LazyAttrTokenStream

A lazy version of AttrTokenStream, which defers creation of an actual AttrTokenStream until it is needed.

NodeRange

A token range within an individual AST node’s (lazy) token stream, i.e. relative to that node’s first token. Distinct from ParserRange so the two kinds of range can’t be mixed up.

ParserRange

A token range within a Parser’s full token stream.

TokenCursor

A TokenStream cursor that produces Tokens. It’s a bit odd that we (a) lex tokens into a nice tree structure (TokenStream), and then (b) use this type to emit them as a linear sequence. But a linear sequence is what the parser expects, for the most part.

TokenStream

A TokenStream is an abstract sequence of tokens, organized into TokenTrees.

TokenStreamIter

TokenTreeCursor

Enums

AttrTokenTree

Like TokenTree, but for AttrTokenStream.

FlatToken 🔒

A helper struct used when building an AttrTokenStream from a LazyAttrTokenStream. Both delimiter and non-delimited tokens are stored as FlatToken::Token. A vector of FlatTokens is then ‘parsed’ to build up an AttrTokenStream with nested AttrTokenTree::Delimited tokens.

LazyAttrTokenStreamInner 🔒

Spacing

Indicates whether a token can join with the following token to form a compound token. Used for conversions to proc_macro::Spacing. Also used to guide pretty-printing, which is where the JointHidden value (which isn’t part of proc_macro::Spacing) comes in useful.

TokenTree

Part of a TokenStream.

Functions

_dummy 🔒

attrs_and_tokens_to_token_trees 🔒

make_attr_token_stream 🔒

Converts a flattened iterator of tokens (including open and close delimiter tokens) into an AttrTokenStream, creating an AttrTokenTree::Delimited for each matching pair of open and close delims.

Type Aliases

NodeReplacement

See the comment on ParserReplacement.

ParserReplacement

Indicates a range of tokens that should be replaced by an AttrsTarget (replacement) or be replaced by nothing (deletion). This is used in two places during token collection.