Module tokenstream

Source
Expand description

§Token Streams

TokenStreams represent syntactic objects before they are converted into ASTs. A TokenStream is, roughly speaking, a sequence of TokenTrees, which are themselves a single Token or a Delimited subsequence of tokens.

§Ownership

TokenStreams are persistent data structures constructed as ropes with reference counted-children. In general, this means that calling an operation on a TokenStream (such as slice) produces an entirely new TokenStream from the borrowed reference to the original. This essentially coerces TokenStreams into “views” of their subparts, and a borrowed TokenStream is sufficient to build an owned TokenStream without taking ownership of the original.

Modules§

size_asserts 🔒

Structs§

AttrTokenStream
An AttrTokenStream is similar to a TokenStream, but with extra information about the tokens for attribute targets. This is used during expansion to perform early cfg-expansion, and to process attributes during proc-macro invocations.
AttrsTarget
Stores the tokens for an attribute target, along with its attributes.
DelimSpacing
DelimSpan
LazyAttrTokenStream
A lazy version of TokenStream, which defers creation of an actual TokenStream until it is needed. Box is here only to reduce the structure size.
TokenStream
A TokenStream is an abstract sequence of tokens, organized into TokenTrees.
TokenStreamIter

Enums§

AttrTokenTree
Like TokenTree, but for AttrTokenStream.
Spacing
Indicates whether a token can join with the following token to form a compound token. Used for conversions to proc_macro::Spacing. Also used to guide pretty-printing, which is where the JointHidden value (which isn’t part of proc_macro::Spacing) comes in useful.
TokenTree
Part of a TokenStream.

Traits§

ToAttrTokenStream

Functions§

_dummy 🔒
attrs_and_tokens_to_token_trees 🔒