TokenStreams represent syntactic objects before they are converted into ASTs.
TokenStream is, roughly speaking, a sequence (eg stream) of
which are themselves a single
Token or a
Delimited subsequence of tokens.
TokenStreams are persistent data structures constructed as ropes with reference
counted-children. In general, this means that calling an operation on a
slice) produces an entirely new
TokenStream from the borrowed reference to
the original. This essentially coerces
TokenStreams into 'views' of their subparts,
and a borrowed
TokenStream is sufficient to build an owned
TokenStream without taking
ownership of the original.
When the main rust parser encounters a syntax-extension invocation, it parses the arguments to the invocation as a token-tree. This is a very loose structure, such that all sorts of different AST-fragments can be passed to syntax extensions using a uniform type.