Struct syntax::tokenstream::TokenStream

pub struct TokenStream(pub Option<Lrc<Vec<TreeAndJoint>>>);

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

Token Streams

A TokenStream is an abstract sequence of tokens, organized into TokenTrees. The goal is for procedural macros to work with TokenStreams and TokenTrees instead of a representation of the abstract syntax tree. Today's TokenTrees can still contain AST via Token::Interpolated for back-compat.

The use of Option is an optimization that avoids the need for an allocation when the stream is empty. However, it is not guaranteed that an empty stream is represented with None; it may be represented as a Some around an empty Vec.

Methods

impl TokenStream

pub(crate) fn add_comma(&self) -> Option<(TokenStream, Span)>

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

Given a TokenStream with a Stream of only two arguments, return a new TokenStream separating the two arguments with a comma for diagnostic suggestions.

impl TokenStream

pub fn len(&self) -> usize

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

pub fn empty() -> TokenStream

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

pub fn is_empty(&self) -> bool

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

pub(crate) fn from_streams(streams: SmallVec<[TokenStream; 2]>) -> TokenStream

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

pub fn new(streams: Vec<TreeAndJoint>) -> TokenStream

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

pub fn append_to_tree_and_joint_vec(self, vec: &mut Vec<TreeAndJoint>)

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

Important traits for Cursor

impl Iterator for Cursor type Item = TokenTree;

pub fn trees(&self) -> Cursor

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

Important traits for Cursor

impl Iterator for Cursor type Item = TokenTree;

pub fn into_trees(self) -> Cursor

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

pub fn eq_unspanned(&self, other: &TokenStream) -> bool

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

Compares two TokenStreams, checking equality without regarding span information.

pub fn probably_equal_for_proc_macro(&self, other: &TokenStream) -> bool

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

pub fn map_enumerated<F: FnMut(usize, TokenTree) -> TokenTree>(
    self,
    f: F
) -> TokenStream

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

pub fn map<F: FnMut(TokenTree) -> TokenTree>(self, f: F) -> TokenStream

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

fn first_tree_and_joint(&self) -> Option<TreeAndJoint>

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

fn last_tree_if_joint(&self) -> Option<TokenTree>

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

Trait Implementations

impl Clone for TokenStream

fn clone(&self) -> TokenStream

Returns a copy of the value.

default fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Eq for TokenStream

default fn assert_receiver_is_total_eq(&self)

impl PartialEq<TokenStream> for TokenStream

fn eq(&self, other: &TokenStream) -> bool

This method tests for self and other values to be equal, and is used by ==.

#[must_use]

default fn ne(&self, other: &Rhs) -> bool

This method tests for !=.

impl From<TokenTree> for TokenStream

fn from(tree: TokenTree) -> TokenStream

Performs the conversion.

impl From<Token> for TokenStream

fn from(token: Token) -> TokenStream

Performs the conversion.

impl Display for TokenStream

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl Debug for TokenStream

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl<T: Into<TokenStream>> FromIterator<T> for TokenStream

fn from_iter<I: IntoIterator<Item = T>>(iter: I) -> Self

Creates a value from an iterator.

impl Encodable for TokenStream

fn encode<E: Encoder>(&self, encoder: &mut E) -> Result<(), E::Error>

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.

impl Decodable for TokenStream

fn decode<D: Decoder>(decoder: &mut D) -> Result<TokenStream, D::Error>

⚙️ This is an internal compiler API. (rustc_private)

This crate is being loaded from the sysroot, a permanently unstable location for private compiler dependencies. It is not intended for general use. Prefer using a public version of this crate from crates.io via Cargo.toml.