TokenStreams represent syntactic objects before they are converted into ASTs.
TokenStream is, roughly speaking, a sequence of
which are themselves a single
Token or a
Delimited subsequence of tokens.
TokenStreams are persistent data structures constructed as ropes with reference
counted-children. In general, this means that calling an operation on a
slice) produces an entirely new
TokenStream from the borrowed reference to
the original. This essentially coerces
TokenStreams into “views” of their subparts,
and a borrowed
TokenStream is sufficient to build an owned
TokenStream without taking
ownership of the original.
AttrAnnotatedTokenStream is similar to a
TokenStream, but with extra
information about the tokens for attribute targets. This is used
during expansion to perform early cfg-expansion, and to process attributes
during proc-macro invocations.
Stores the tokens for an attribute target, along with its attributes.
A lazy version of
TokenStream, which defers creation
of an actual
TokenStream until it is needed.
Box is here only to reduce the structure size.
TokenTree, but for
When the main Rust parser encounters a syntax-extension invocation, it parses the arguments to the invocation as a token tree. This is a very loose structure, such that all sorts of different AST fragments can be passed to syntax extensions using a uniform type.