Module rustc_ast::tokenstream [−][src]
Expand description
Token Streams
TokenStreams represent syntactic objects before they are converted into ASTs.
A TokenStream is, roughly speaking, a sequence of TokenTrees,
which are themselves a single Token or a Delimited subsequence of tokens.
Ownership
TokenStreams are persistent data structures constructed as ropes with reference
counted-children. In general, this means that calling an operation on a TokenStream
(such as slice) produces an entirely new TokenStream from the borrowed reference to
the original. This essentially coerces TokenStreams into “views” of their subparts,
and a borrowed TokenStream is sufficient to build an owned TokenStream without taking
ownership of the original.
Structs
A AttrAnnotatedTokenStream is similar to a TokenStream, but with extra
information about the tokens for attribute targets. This is used
during expansion to perform early cfg-expansion, and to process attributes
during proc-macro invocations.
Stores the tokens for an attribute target, along with its attributes.
Owning by-value iterator over a TokenStream.
By-reference iterator over a TokenStream.
A lazy version of TokenStream, which defers creation
of an actual TokenStream until it is needed.
Box is here only to reduce the structure size.
A TokenStream is an abstract sequence of tokens, organized into TokenTrees.
Enums
Like TokenTree, but for AttrAnnotatedTokenStream
When the main Rust parser encounters a syntax-extension invocation, it parses the arguments to the invocation as a token tree. This is a very loose structure, such that all sorts of different AST fragments can be passed to syntax extensions using a uniform type.