Expand description
Manually manage memory through raw pointers.
See also the pointer primitive types.
§Safety
Many functions in this module take raw pointers as arguments and read from or write to them. For
this to be safe, these pointers must be valid for the given access. Whether a pointer is valid
depends on the operation it is used for (read or write), and the extent of the memory that is
accessed (i.e., how many bytes are read/written) – it makes no sense to ask “is this pointer
valid”; one has to ask “is this pointer valid for a given access”. Most functions use *mut T
and *const T
to access only a single value, in which case the documentation omits the size and
implicitly assumes it to be size_of::<T>()
bytes.
The precise rules for validity are not determined yet. The guarantees that are provided at this point are very minimal:
- For operations of size zero, every pointer is valid, including the null pointer. The following points are only concerned with non-zero-sized accesses.
- A null pointer is never valid.
- For a pointer to be valid, it is necessary, but not always sufficient, that the pointer be dereferenceable: the memory range of the given size starting at the pointer must all be within the bounds of a single allocated object. Note that in Rust, every (stack-allocated) variable is considered a separate allocated object.
- All accesses performed by functions in this module are non-atomic in the sense
of atomic operations used to synchronize between threads. This means it is
undefined behavior to perform two concurrent accesses to the same location from different
threads unless both accesses only read from memory. Notice that this explicitly
includes
read_volatile
andwrite_volatile
: Volatile accesses cannot be used for inter-thread synchronization. - The result of casting a reference to a pointer is valid for as long as the underlying object is live and no reference (just raw pointers) is used to access the same memory. That is, reference and pointer accesses cannot be interleaved.
These axioms, along with careful use of offset
for pointer arithmetic,
are enough to correctly implement many useful things in unsafe code. Stronger guarantees
will be provided eventually, as the aliasing rules are being determined. For more
information, see the book as well as the section in the reference devoted
to undefined behavior.
We say that a pointer is “dangling” if it is not valid for any non-zero-sized accesses. This
means out-of-bounds pointers, pointers to freed memory, null pointers, and pointers created with
NonNull::dangling
are all dangling.
§Alignment
Valid raw pointers as defined above are not necessarily properly aligned (where
“proper” alignment is defined by the pointee type, i.e., *const T
must be
aligned to mem::align_of::<T>()
). However, most functions require their
arguments to be properly aligned, and will explicitly state
this requirement in their documentation. Notable exceptions to this are
read_unaligned
and write_unaligned
.
When a function requires proper alignment, it does so even if the access
has size 0, i.e., even if memory is not actually touched. Consider using
NonNull::dangling
in such cases.
§Allocated object
An allocated object is a subset of program memory which is addressable
from Rust, and within which pointer arithmetic is possible. Examples of
allocated objects include heap allocations, stack-allocated variables,
statics, and consts. The safety preconditions of some Rust operations -
such as offset
and field projections (expr.field
) - are defined in
terms of the allocated objects on which they operate.
An allocated object has a base address, a size, and a set of memory addresses. It is possible for an allocated object to have zero size, but such an allocated object will still have a base address. The base address of an allocated object is not necessarily unique. While it is currently the case that an allocated object always has a set of memory addresses which is fully contiguous (i.e., has no “holes”), there is no guarantee that this will not change in the future.
For any allocated object with base
address, size
, and a set of
addresses
, the following are guaranteed:
- For all addresses
a
inaddresses
,a
is in the rangebase .. (base + size)
(note that this requiresa < base + size
, nota <= base + size
) base
is not equal tonull()
(i.e., the address with the numerical value 0)base + size <= usize::MAX
size <= isize::MAX
As a consequence of these guarantees, given any address a
within the set
of addresses of an allocated object:
- It is guaranteed that
a - base
does not overflowisize
- It is guaranteed that
a - base
is non-negative - It is guaranteed that, given
o = a - base
(i.e., the offset ofa
within the allocated object),base + o
will not wrap around the address space (in other words, will not overflowusize
)
§Strict Provenance
The following text is non-normative, insufficiently formal, and is an extremely strict interpretation of provenance. It’s ok if your code doesn’t strictly conform to it.
Strict Provenance is an experimental set of APIs that help tools that try to validate the memory-safety of your program’s execution. Notably this includes Miri and CHERI, which can detect when you access out of bounds memory or otherwise violate Rust’s memory model.
Provenance must exist in some form for any programming language compiled for modern computer architectures, but specifying a model for provenance in a way that is useful to both compilers and programmers is an ongoing challenge. The Strict Provenance experiment seeks to explore the question: what if we just said you couldn’t do all the nasty operations that make provenance so messy?
What APIs would have to be removed? What APIs would have to be added? How much would code have to change, and is it worse or better now? Would any patterns become truly inexpressible? Could we carve out special exceptions for those patterns? Should we?
A secondary goal of this project is to see if we can disambiguate the many functions of
pointer<->integer casts enough for the definition of usize
to be loosened so that it
isn’t pointer-sized but address-space/offset/allocation-sized (we’ll probably continue
to conflate these notions). This would potentially make it possible to more efficiently
target platforms where pointers are larger than offsets, such as CHERI and maybe some
segmented architectures.
§Provenance
This section is non-normative and is part of the Strict Provenance experiment.
Pointers are not simply an “integer” or “address”. For instance, it’s uncontroversial to say that a Use After Free is clearly Undefined Behaviour, even if you “get lucky” and the freed memory gets reallocated before your read/write (in fact this is the worst-case scenario, UAFs would be much less concerning if this didn’t happen!). To rationalize this claim, pointers need to somehow be more than just their addresses: they must have provenance.
When an allocation is created, that allocation has a unique Original Pointer. For alloc APIs this is literally the pointer the call returns, and for local variables and statics, this is the name of the variable/static. This is mildly overloading the term “pointer” for the sake of brevity/exposition.
The Original Pointer for an allocation is guaranteed to have unique access to the entire allocation and only that allocation. In this sense, an allocation can be thought of as a “sandbox” that cannot be broken into or out of. Provenance is the permission to access an allocation’s sandbox and has both a spatial and temporal component:
- Spatial: A range of bytes that the pointer is allowed to access.
- Temporal: The lifetime (of the allocation) that access to these bytes is tied to.
Spatial provenance makes sure you don’t go beyond your sandbox, while temporal provenance makes sure that you can’t “get lucky” after your permission to access some memory has been revoked (either through deallocations or borrows expiring).
Provenance is implicitly shared with all pointers transitively derived from
The Original Pointer through operations like offset
, borrowing, and pointer casts.
Some operations may shrink the derived provenance, limiting how much memory it can
access or how long it’s valid for (i.e. borrowing a subfield and subslicing).
Shrinking provenance cannot be undone: even if you “know” there is a larger allocation, you
can’t derive a pointer with a larger provenance. Similarly, you cannot “recombine”
two contiguous provenances back into one (i.e. with a fn merge(&[T], &[T]) -> &[T]
).
A reference to a value always has provenance over exactly the memory that field occupies. A reference to a slice always has provenance over exactly the range that slice describes.
If an allocation is deallocated, all pointers with provenance to that allocation become invalidated, and effectively lose their provenance.
The strict provenance experiment is mostly only interested in exploring stricter spatial provenance. In this sense it can be thought of as a subset of the more ambitious and formal Stacked Borrows research project, which is what tools like Miri are based on. In particular, Stacked Borrows is necessary to properly describe what borrows are allowed to do and when they become invalidated. This necessarily involves much more complex temporal reasoning than simply identifying allocations. Adjusting APIs and code for the strict provenance experiment will also greatly help Stacked Borrows.
§Pointer Vs Addresses
This section is non-normative and is part of the Strict Provenance experiment.
One of the largest historical issues with trying to define provenance is that programmers freely convert between pointers and integers. Once you allow for this, it generally becomes impossible to accurately track and preserve provenance information, and you need to appeal to very complex and unreliable heuristics. But of course, converting between pointers and integers is very useful, so what can we do?
Also did you know WASM is actually a “Harvard Architecture”? As in function pointers are handled completely differently from data pointers? And we kind of just shipped Rust on WASM without really addressing the fact that we let you freely convert between function pointers and data pointers, because it mostly Just Works? Let’s just put that on the “pointer casts are dubious” pile.
Strict Provenance attempts to square these circles by decoupling Rust’s traditional conflation
of pointers and usize
(and isize
), and defining a pointer to semantically contain the
following information:
- The address-space it is part of (e.g. “data” vs “code” in WASM).
- The address it points to, which can be represented by a
usize
. - The provenance it has, defining the memory it has permission to access. Provenance can be absent, in which case the pointer does not have permission to access any memory.
Under Strict Provenance, a usize cannot accurately represent a pointer, and converting from a pointer to a usize is generally an operation which only extracts the address. It is therefore impossible to construct a valid pointer from a usize because there is no way to restore the address-space and provenance. In other words, pointer-integer-pointer roundtrips are not possible (in the sense that the resulting pointer is not dereferenceable).
The key insight to making this model at all viable is the with_addr
method:
/// Creates a new pointer with the given address.
///
/// This performs the same operation as an `addr as ptr` cast, but copies
/// the *address-space* and *provenance* of `self` to the new pointer.
/// This allows us to dynamically preserve and propagate this important
/// information in a way that is otherwise impossible with a unary cast.
///
/// This is equivalent to using `wrapping_offset` to offset `self` to the
/// given address, and therefore has all the same capabilities and restrictions.
pub fn with_addr(self, addr: usize) -> Self;
So you’re still able to drop down to the address representation and do whatever
clever bit tricks you want as long as you’re able to keep around a pointer
into the allocation you care about that can “reconstitute” the other parts of the pointer.
Usually this is very easy, because you only are taking a pointer, messing with the address,
and then immediately converting back to a pointer. To make this use case more ergonomic,
we provide the map_addr
method.
To help make it clear that code is “following” Strict Provenance semantics, we also provide an
addr
method which promises that the returned address is not part of a
pointer-usize-pointer roundtrip. In the future we may provide a lint for pointer<->integer
casts to help you audit if your code conforms to strict provenance.
§Using Strict Provenance
Most code needs no changes to conform to strict provenance, as the only really concerning operation that wasn’t obviously already Undefined Behaviour is casts from usize to a pointer. For code which does cast a usize to a pointer, the scope of the change depends on exactly what you’re doing.
In general, you just need to make sure that if you want to convert a usize address to a pointer and then use that pointer to read/write memory, you need to keep around a pointer that has sufficient provenance to perform that read/write itself. In this way all of your casts from an address to a pointer are essentially just applying offsets/indexing.
This is generally trivial to do for simple cases like tagged pointers as long as you represent the tagged pointer as an actual pointer and not a usize. For instance:
#![feature(strict_provenance)]
unsafe {
// A flag we want to pack into our pointer
static HAS_DATA: usize = 0x1;
static FLAG_MASK: usize = !HAS_DATA;
// Our value, which must have enough alignment to have spare least-significant-bits.
let my_precious_data: u32 = 17;
assert!(core::mem::align_of::<u32>() > 1);
// Create a tagged pointer
let ptr = &my_precious_data as *const u32;
let tagged = ptr.map_addr(|addr| addr | HAS_DATA);
// Check the flag:
if tagged.addr() & HAS_DATA != 0 {
// Untag and read the pointer
let data = *tagged.map_addr(|addr| addr & FLAG_MASK);
assert_eq!(data, 17);
} else {
unreachable!()
}
}
Run(Yes, if you’ve been using AtomicUsize for pointers in concurrent datastructures, you should be using AtomicPtr instead. If that messes up the way you atomically manipulate pointers, we would like to know why, and what needs to be done to fix it.)
Something more complicated and just generally evil like an XOR-List requires more significant changes like allocating all nodes in a pre-allocated Vec or Arena and using a pointer to the whole allocation to reconstitute the XORed addresses.
Situations where a valid pointer must be created from just an address, such as baremetal code accessing a memory-mapped interface at a fixed address, are an open question on how to support. These situations will still be allowed, but we might require some kind of “I know what I’m doing” annotation to explain the situation to the compiler. It’s also possible they need no special attention at all, because they’re generally accessing memory outside the scope of “the abstract machine”, or already using “I know what I’m doing” annotations like “volatile”.
Under Strict Provenance it is Undefined Behaviour to:
-
Access memory through a pointer that does not have provenance over that memory.
-
offset
a pointer to or from an address it doesn’t have provenance over. This means it’s always UB to offset a pointer derived from something deallocated, even if the offset is 0. Note that a pointer “one past the end” of its provenance is not actually outside its provenance, it just has 0 bytes it can load/store.
But it is still sound to:
-
Create a pointer without provenance from just an address (see
ptr::dangling
). Such a pointer cannot be used for memory accesses (except for zero-sized accesses). This can still be useful for sentinel values likenull
or to represent a tagged pointer that will never be dereferenceable. In general, it is always sound for an integer to pretend to be a pointer “for fun” as long as you don’t use operations on it which require it to be valid (non-zero-sized offset, read, write, etc). -
Forge an allocation of size zero at any sufficiently aligned non-null address. i.e. the usual “ZSTs are fake, do what you want” rules apply but this only applies for actual forgery (integers cast to pointers). If you borrow some struct’s field that happens to be zero-sized, the resulting pointer will have provenance tied to that allocation, and it will still get invalidated if the allocation gets deallocated. In the future we may introduce an API to make such a forged allocation explicit.
-
wrapping_offset
a pointer outside its provenance. This includes pointers which have “no” provenance. Unfortunately there may be practical limits on this for a particular platform, and it’s an open question as to how to specify this (if at all). Notably, CHERI relies on a compression scheme that can’t handle a pointer getting offset “too far” out of bounds. If this happens, the address returned byaddr
will be the value you expect, but the provenance will get invalidated and using it to read/write will fault. The details of this are architecture-specific and based on alignment, but the buffer on either side of the pointer’s range is pretty generous (think kilobytes, not bytes). -
Compare arbitrary pointers by address. Addresses are just integers and so there is always a coherent answer, even if the pointers are dangling or from different address-spaces/provenances. Of course, comparing addresses from different address-spaces is generally going to be meaningless, but so is comparing Kilograms to Meters, and Rust doesn’t prevent that either. Similarly, if you get “lucky” and notice that a pointer one-past-the-end is the “same” address as the start of an unrelated allocation, anything you do with that fact is probably going to be gibberish. The scope of that gibberish is kept under control by the fact that the two pointers still aren’t allowed to access the other’s allocation (bytes), because they still have different provenance.
-
Perform pointer tagging tricks. This falls out of
wrapping_offset
but is worth mentioning in more detail because of the limitations of CHERI. Low-bit tagging is very robust, and often doesn’t even go out of bounds because types ensure size >= align (and over-aligning actually gives CHERI more flexibility). Anything more complex than this rapidly enters “extremely platform-specific” territory as certain things may or may not be allowed based on specific supported operations. For instance, ARM explicitly supports high-bit tagging, and so CHERI on ARM inherits that and should support it.
§Exposed Provenance
This section is non-normative and is an extension to the Strict Provenance experiment.
As discussed above, pointer-usize-pointer roundtrips are not possible under Strict Provenance. This is by design: the goal of Strict Provenance is to provide a clear specification that we are confident can be formalized unambiguously and can be subject to precise formal reasoning.
However, there exist situations where pointer-usize-pointer roundtrips cannot be avoided, or
where avoiding them would require major refactoring. Legacy platform APIs also regularly assume
that usize
can capture all the information that makes up a pointer. The goal of Strict
Provenance is not to rule out such code; the goal is to put all the other pointer-manipulating
code onto a more solid foundation. Strict Provenance is about improving the situation where
possible (all the code that can be written with Strict Provenance) without making things worse
for situations where Strict Provenance is insufficient.
For these situations, there is a highly experimental extension to Strict Provenance called Exposed Provenance. This extension permits pointer-usize-pointer roundtrips. However, its semantics are on much less solid footing than Strict Provenance, and at this point it is not yet clear where a satisfying unambiguous semantics can be defined for Exposed Provenance. Furthermore, Exposed Provenance will not work (well) with tools like Miri and CHERI.
Exposed Provenance is provided by the expose_provenance
and with_exposed_provenance
methods,
which are meant to replace as
casts between pointers and integers. expose_provenance
is a lot like
addr
, but additionally adds the provenance of the pointer to a global list of ‘exposed’
provenances. (This list is purely conceptual, it exists for the purpose of specifying Rust but
is not materialized in actual executions, except in tools like Miri.) with_exposed_provenance
can be used to construct a pointer with one of these previously ‘exposed’ provenances.
with_exposed_provenance
takes only addr: usize
as arguments, so unlike in with_addr
there is
no indication of what the correct provenance for the returned pointer is – and that is exactly
what makes pointer-usize-pointer roundtrips so tricky to rigorously specify! There is no
algorithm that decides which provenance will be used. You can think of this as “guessing” the
right provenance, and the guess will be “maximally in your favor”, in the sense that if there is
any way to avoid undefined behavior, then that is the guess that will be taken. However, if
there is no previously ‘exposed’ provenance that justifies the way the returned pointer will
be used, the program has undefined behavior.
Using expose_provenance
or with_exposed_provenance
(or the as
casts) means that code is
not following Strict Provenance rules. The goal of the Strict Provenance experiment is to
determine how far one can get in Rust without the use of expose_provenance
and
with_exposed_provenance
, and to encourage code to be written with Strict Provenance APIs only.
Maximizing the amount of such code is a major win for avoiding specification complexity and to
facilitate adoption of tools like CHERI and Miri that can be a big help in increasing the
confidence in (unsafe) Rust code.
Macros§
- Create a
const
raw pointer to a place, without creating an intermediate reference. - Create a
mut
raw pointer to a place, without creating an intermediate reference.
Structs§
*mut T
but non-zero and covariant.- AlignmentExperimentalA type storing a
usize
which is a power of two, and thus represents a possible alignment in the Rust abstract machine. - DynMetadataExperimentalThe metadata for a
Dyn = dyn SomeTrait
trait object type.
Traits§
- PointeeExperimentalProvides the pointer metadata type of any pointed-to type.
Functions§
- Compares the addresses of the two pointers for equality, ignoring any metadata in fat pointers.
- copy⚠Copies
count * size_of::<T>()
bytes fromsrc
todst
. The source and destination may overlap. - Copies
count * size_of::<T>()
bytes fromsrc
todst
. The source and destination must not overlap. - Executes the destructor (if any) of the pointed-to value.
- Compares raw pointers for equality.
- Convert a mutable reference to a raw pointer.
- Convert a reference to a raw pointer.
- Hash a raw pointer.
- Creates a null raw pointer.
- Creates a null mutable raw pointer.
- read⚠Reads the value from
src
without moving it. This leaves the memory insrc
unchanged. - Reads the value from
src
without moving it. This leaves the memory insrc
unchanged. - Performs a volatile read of the value from
src
without moving it. This leaves the memory insrc
unchanged. - Moves
src
into the pointeddst
, returning the previousdst
value. - Forms a raw slice from a pointer and a length.
- Forms a raw mutable slice from a pointer and a length.
- swap⚠Swaps the values at two mutable locations of the same type, without deinitializing either.
- Swaps
count * size_of::<T>()
bytes between the two regions of memory beginning atx
andy
. The two regions must not overlap. - Overwrites a memory location with the given value without reading or dropping the old value.
- Sets
count * size_of::<T>()
bytes of memory starting atdst
toval
. - Overwrites a memory location with the given value without reading or dropping the old value.
- Performs a volatile write of a memory location with the given value without reading or dropping the old value.
- danglingExperimentalCreates a new pointer that is dangling, but well-aligned.
- dangling_mutExperimentalCreates a new pointer that is dangling, but well-aligned.
- from_raw_partsExperimentalForms a (possibly-wide) raw pointer from a data pointer and metadata.
- from_raw_parts_mutExperimentalPerforms the same functionality as
from_raw_parts
, except that a raw*mut
pointer is returned, as opposed to a raw*const
pointer. - metadataExperimentalExtract the metadata component of a pointer.
- with_exposed_provenanceExperimentalConvert an address back to a pointer, picking up a previously ‘exposed’ provenance.
- with_exposed_provenance_mutExperimentalConvert an address back to a mutable pointer, picking up a previously ‘exposed’ provenance.
- without_provenanceExperimentalCreates a pointer with the given address and no provenance.
- without_provenance_mutExperimentalCreates a pointer with the given address and no provenance.