TokenStreams represent syntactic objects before they are converted into ASTs.
TokenStream is, roughly speaking, a sequence of
which are themselves a single
Token or a
Delimited subsequence of tokens.
TokenStreams are persistent data structures constructed as ropes with reference
counted-children. In general, this means that calling an operation on a
slice) produces an entirely new
TokenStream from the borrowed reference to
the original. This essentially coerces
TokenStreams into “views” of their subparts,
and a borrowed
TokenStream is sufficient to build an owned
TokenStream without taking
ownership of the original.
AttrTokenStreamis similar to a
TokenStream, but with extra information about the tokens for attribute targets. This is used during expansion to perform early cfg-expansion, and to process attributes during proc-macro invocations.
TokenStream, which defers creation of an actual
TokenStreamuntil it is needed.
Boxis here only to reduce the structure size.
TokenTree, but for
proc_macro::Spacing, but for tokens.