Module tokenstream 

Token Streams

TokenStreams represent syntactic objects before they are converted into ASTs. A TokenStream is, roughly speaking, a sequence of TokenTrees, which are themselves a single Token or a Delimited subsequence of tokens.

Re-exports

use std::borrow::Cow; 🔒

use std::hash::Hash; 🔒

use std::hash::Hash; 🔒

use std::ops::Range; 🔒

use std::sync::Arc; 🔒

use std::sync::LazyLock; 🔒

use std::cmp; 🔒

use std::fmt; 🔒

use std::iter; 🔒

use std::mem; 🔒

use rustc_data_structures::stable_hash::StableHash; 🔒

use rustc_data_structures::stable_hash::StableHashCtxt; 🔒

use rustc_data_structures::stable_hash::StableHasher; 🔒

use rustc_data_structures::sync; 🔒

use rustc_macros::Decodable; 🔒

use rustc_macros::Encodable; 🔒

use rustc_macros::StableHash; 🔒

use rustc_macros::Walkable; 🔒

use rustc_serialize::Decodable; 🔒

use rustc_serialize::Encodable; 🔒

use rustc_span::DUMMY_SP; 🔒

use rustc_span::Span; 🔒

use rustc_span::SpanDecoder; 🔒

use rustc_span::SpanEncoder; 🔒

use rustc_span::Symbol; 🔒

use rustc_span::sym; 🔒

use thin_vec::ThinVec; 🔒

use crate::ast::AttrStyle; 🔒

use crate::ast_traits::HasTokens; 🔒

use crate::token; 🔒

use crate::token::Delimiter; 🔒

use crate::token::Token; 🔒

use crate::token::TokenKind; 🔒

use crate::AttrVec; 🔒

use crate::Attribute; 🔒

Modules

size_asserts 🔒

Structs

AttrsTarget

Stores the tokens for an attribute target, along with its attributes.

DelimSpacing

DelimSpan

LazyTokenStream

A lazy version of TokenStream, which defers creation of an actual TokenStream until it is needed.

NodeRange

A token range within an individual AST node’s (lazy) token stream, i.e. relative to that node’s first token. Distinct from ParserRange so the two kinds of range can’t be mixed up.

ParserRange

A token range within a Parser’s full token stream.

TokenCursor

A TokenStream cursor that produces Tokens. It’s a bit odd that we (a) lex tokens into a nice tree structure (TokenStream), and then (b) use this type to emit them as a linear sequence. But a linear sequence is what the parser expects, for the most part.

TokenStream

A TokenStream is an abstract sequence of tokens, organized into TokenTrees.

TokenStreamIter

TokenTreeCursor 🔒

WithTokens

Enums

FlatToken 🔒

A helper struct used when building a TokenStream from a LazyTokenStream. Both delimiter and non-delimited tokens are stored as FlatToken::Token. A vector of FlatTokens is then ‘parsed’ to build up a TokenStream with nested TokenTree::Delimited tokens.

LazyTokenStreamInner 🔒

Spacing

Indicates whether a token can join with the following token to form a compound token. Used for conversions to proc_macro::Spacing. Also used to guide pretty-printing, which is where the JointHidden value (which isn’t part of proc_macro::Spacing) comes in useful.

TokenTree

Part of a TokenStream.

Statics

EMPTY_TOKEN_STREAM 🔒

A shared empty token stream, used to avoid an unnecessary Arc allocation for every empty token stream.

Functions

_dummy 🔒

append_attrs_and_stream_as_token_trees_to 🔒

append_flat_tokens_as_token_trees_to 🔒

Converts a flattened iterator of tokens (including open and close delimiter tokens) into a sequence of TokenTree (appended to res), creating a TokenTree::Delimited for each matching pair of open and close delims. Any FlatToken::AttrsTarget element will only be appended if configure succeeds, and configure can also modify any such element.

keep_all 🔒

A ConfigureFn that keeps the target, i.e. the node and its tokens.

Type Aliases

ConfigureFn 🔒

Fn type used to do (or not do) cfg expansion. It can modify attributes. The return value indicates if the target should be retained. Any children of discarded targets are skipped.

NodeReplacement

See the comment on ParserReplacement.

ParserReplacement

Indicates a range of tokens that should be replaced by an AttrsTarget (replacement) or be replaced by nothing (deletion). This is used in two places during token collection.