1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478
//! The memory subsystem.
//!
//! Generally, we use `Pointer` to denote memory addresses. However, some operations
//! have a "size"-like parameter, and they take `Scalar` for the address because
//! if the size is 0, then the pointer can also be a (properly aligned, non-null)
//! integer. It is crucial that these operations call `check_align` *before*
//! short-circuiting the empty case!
use std::assert_matches::assert_matches;
use std::borrow::Cow;
use std::cell::Cell;
use std::collections::VecDeque;
use std::{fmt, ptr};
use rustc_ast::Mutability;
use rustc_data_structures::fx::{FxHashSet, FxIndexMap};
use rustc_hir::def::DefKind;
use rustc_middle::bug;
use rustc_middle::mir::display_allocation;
use rustc_middle::ty::{self, Instance, ParamEnv, Ty, TyCtxt};
use rustc_target::abi::{Align, HasDataLayout, Size};
use tracing::{debug, instrument, trace};
use super::{
alloc_range, err_ub, err_ub_custom, throw_ub, throw_ub_custom, throw_unsup, throw_unsup_format,
AllocBytes, AllocId, AllocMap, AllocRange, Allocation, CheckAlignMsg, CheckInAllocMsg,
CtfeProvenance, GlobalAlloc, InterpCx, InterpResult, Machine, MayLeak, Misalignment, Pointer,
PointerArithmetic, Provenance, Scalar,
};
use crate::fluent_generated as fluent;
#[derive(Debug, PartialEq, Copy, Clone)]
pub enum MemoryKind<T> {
/// Stack memory. Error if deallocated except during a stack pop.
Stack,
/// Memory allocated by `caller_location` intrinsic. Error if ever deallocated.
CallerLocation,
/// Additional memory kinds a machine wishes to distinguish from the builtin ones.
Machine(T),
}
impl<T: MayLeak> MayLeak for MemoryKind<T> {
#[inline]
fn may_leak(self) -> bool {
match self {
MemoryKind::Stack => false,
MemoryKind::CallerLocation => true,
MemoryKind::Machine(k) => k.may_leak(),
}
}
}
impl<T: fmt::Display> fmt::Display for MemoryKind<T> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
MemoryKind::Stack => write!(f, "stack variable"),
MemoryKind::CallerLocation => write!(f, "caller location"),
MemoryKind::Machine(m) => write!(f, "{m}"),
}
}
}
/// The return value of `get_alloc_info` indicates the "kind" of the allocation.
#[derive(Copy, Clone, PartialEq, Debug)]
pub enum AllocKind {
/// A regular live data allocation.
LiveData,
/// A function allocation (that fn ptrs point to).
Function,
/// A (symbolic) vtable allocation.
VTable,
/// A dead allocation.
Dead,
}
/// The value of a function pointer.
#[derive(Debug, Copy, Clone)]
pub enum FnVal<'tcx, Other> {
Instance(Instance<'tcx>),
Other(Other),
}
impl<'tcx, Other> FnVal<'tcx, Other> {
pub fn as_instance(self) -> InterpResult<'tcx, Instance<'tcx>> {
match self {
FnVal::Instance(instance) => Ok(instance),
FnVal::Other(_) => {
throw_unsup_format!("'foreign' function pointers are not supported in this context")
}
}
}
}
// `Memory` has to depend on the `Machine` because some of its operations
// (e.g., `get`) call a `Machine` hook.
pub struct Memory<'tcx, M: Machine<'tcx>> {
/// Allocations local to this instance of the interpreter. The kind
/// helps ensure that the same mechanism is used for allocation and
/// deallocation. When an allocation is not found here, it is a
/// global and looked up in the `tcx` for read access. Some machines may
/// have to mutate this map even on a read-only access to a global (because
/// they do pointer provenance tracking and the allocations in `tcx` have
/// the wrong type), so we let the machine override this type.
/// Either way, if the machine allows writing to a global, doing so will
/// create a copy of the global allocation here.
// FIXME: this should not be public, but interning currently needs access to it
pub(super) alloc_map: M::MemoryMap,
/// Map for "extra" function pointers.
extra_fn_ptr_map: FxIndexMap<AllocId, M::ExtraFnVal>,
/// To be able to compare pointers with null, and to check alignment for accesses
/// to ZSTs (where pointers may dangle), we keep track of the size even for allocations
/// that do not exist any more.
// FIXME: this should not be public, but interning currently needs access to it
pub(super) dead_alloc_map: FxIndexMap<AllocId, (Size, Align)>,
/// This stores whether we are currently doing reads purely for the purpose of validation.
/// Those reads do not trigger the machine's hooks for memory reads.
/// Needless to say, this must only be set with great care!
validation_in_progress: Cell<bool>,
}
/// A reference to some allocation that was already bounds-checked for the given region
/// and had the on-access machine hooks run.
#[derive(Copy, Clone)]
pub struct AllocRef<'a, 'tcx, Prov: Provenance, Extra, Bytes: AllocBytes = Box<[u8]>> {
alloc: &'a Allocation<Prov, Extra, Bytes>,
range: AllocRange,
tcx: TyCtxt<'tcx>,
alloc_id: AllocId,
}
/// A reference to some allocation that was already bounds-checked for the given region
/// and had the on-access machine hooks run.
pub struct AllocRefMut<'a, 'tcx, Prov: Provenance, Extra, Bytes: AllocBytes = Box<[u8]>> {
alloc: &'a mut Allocation<Prov, Extra, Bytes>,
range: AllocRange,
tcx: TyCtxt<'tcx>,
alloc_id: AllocId,
}
impl<'tcx, M: Machine<'tcx>> Memory<'tcx, M> {
pub fn new() -> Self {
Memory {
alloc_map: M::MemoryMap::default(),
extra_fn_ptr_map: FxIndexMap::default(),
dead_alloc_map: FxIndexMap::default(),
validation_in_progress: Cell::new(false),
}
}
/// This is used by [priroda](https://github.com/oli-obk/priroda)
pub fn alloc_map(&self) -> &M::MemoryMap {
&self.alloc_map
}
}
impl<'tcx, M: Machine<'tcx>> InterpCx<'tcx, M> {
/// Call this to turn untagged "global" pointers (obtained via `tcx`) into
/// the machine pointer to the allocation. Must never be used
/// for any other pointers, nor for TLS statics.
///
/// Using the resulting pointer represents a *direct* access to that memory
/// (e.g. by directly using a `static`),
/// as opposed to access through a pointer that was created by the program.
///
/// This function can fail only if `ptr` points to an `extern static`.
#[inline]
pub fn global_root_pointer(
&self,
ptr: Pointer<CtfeProvenance>,
) -> InterpResult<'tcx, Pointer<M::Provenance>> {
let alloc_id = ptr.provenance.alloc_id();
// We need to handle `extern static`.
match self.tcx.try_get_global_alloc(alloc_id) {
Some(GlobalAlloc::Static(def_id)) if self.tcx.is_thread_local_static(def_id) => {
// Thread-local statics do not have a constant address. They *must* be accessed via
// `ThreadLocalRef`; we can never have a pointer to them as a regular constant value.
bug!("global memory cannot point to thread-local static")
}
Some(GlobalAlloc::Static(def_id)) if self.tcx.is_foreign_item(def_id) => {
return M::extern_static_pointer(self, def_id);
}
None => {
assert!(
self.memory.extra_fn_ptr_map.contains_key(&alloc_id),
"{alloc_id:?} is neither global nor a function pointer"
);
}
_ => {}
}
// And we need to get the provenance.
M::adjust_alloc_root_pointer(self, ptr, M::GLOBAL_KIND.map(MemoryKind::Machine))
}
pub fn fn_ptr(&mut self, fn_val: FnVal<'tcx, M::ExtraFnVal>) -> Pointer<M::Provenance> {
let id = match fn_val {
FnVal::Instance(instance) => {
let salt = M::get_global_alloc_salt(self, Some(instance));
self.tcx.reserve_and_set_fn_alloc(instance, salt)
}
FnVal::Other(extra) => {
// FIXME(RalfJung): Should we have a cache here?
let id = self.tcx.reserve_alloc_id();
let old = self.memory.extra_fn_ptr_map.insert(id, extra);
assert!(old.is_none());
id
}
};
// Functions are global allocations, so make sure we get the right root pointer.
// We know this is not an `extern static` so this cannot fail.
self.global_root_pointer(Pointer::from(id)).unwrap()
}
pub fn allocate_ptr(
&mut self,
size: Size,
align: Align,
kind: MemoryKind<M::MemoryKind>,
) -> InterpResult<'tcx, Pointer<M::Provenance>> {
let alloc = if M::PANIC_ON_ALLOC_FAIL {
Allocation::uninit(size, align)
} else {
Allocation::try_uninit(size, align)?
};
self.allocate_raw_ptr(alloc, kind)
}
pub fn allocate_bytes_ptr(
&mut self,
bytes: &[u8],
align: Align,
kind: MemoryKind<M::MemoryKind>,
mutability: Mutability,
) -> InterpResult<'tcx, Pointer<M::Provenance>> {
let alloc = Allocation::from_bytes(bytes, align, mutability);
self.allocate_raw_ptr(alloc, kind)
}
pub fn allocate_raw_ptr(
&mut self,
alloc: Allocation<M::Provenance, (), M::Bytes>,
kind: MemoryKind<M::MemoryKind>,
) -> InterpResult<'tcx, Pointer<M::Provenance>> {
let id = self.tcx.reserve_alloc_id();
debug_assert_ne!(
Some(kind),
M::GLOBAL_KIND.map(MemoryKind::Machine),
"dynamically allocating global memory"
);
// We have set things up so we don't need to call `adjust_from_tcx` here,
// so we avoid copying the entire allocation contents.
let extra = M::init_alloc_extra(self, id, kind, alloc.size(), alloc.align)?;
let alloc = alloc.with_extra(extra);
self.memory.alloc_map.insert(id, (kind, alloc));
M::adjust_alloc_root_pointer(self, Pointer::from(id), Some(kind))
}
pub fn reallocate_ptr(
&mut self,
ptr: Pointer<Option<M::Provenance>>,
old_size_and_align: Option<(Size, Align)>,
new_size: Size,
new_align: Align,
kind: MemoryKind<M::MemoryKind>,
) -> InterpResult<'tcx, Pointer<M::Provenance>> {
let (alloc_id, offset, _prov) = self.ptr_get_alloc_id(ptr, 0)?;
if offset.bytes() != 0 {
throw_ub_custom!(
fluent::const_eval_realloc_or_alloc_with_offset,
ptr = format!("{ptr:?}"),
kind = "realloc"
);
}
// For simplicities' sake, we implement reallocate as "alloc, copy, dealloc".
// This happens so rarely, the perf advantage is outweighed by the maintenance cost.
let new_ptr = self.allocate_ptr(new_size, new_align, kind)?;
let old_size = match old_size_and_align {
Some((size, _align)) => size,
None => self.get_alloc_raw(alloc_id)?.size(),
};
// This will also call the access hooks.
self.mem_copy(ptr, new_ptr.into(), old_size.min(new_size), /*nonoverlapping*/ true)?;
self.deallocate_ptr(ptr, old_size_and_align, kind)?;
Ok(new_ptr)
}
#[instrument(skip(self), level = "debug")]
pub fn deallocate_ptr(
&mut self,
ptr: Pointer<Option<M::Provenance>>,
old_size_and_align: Option<(Size, Align)>,
kind: MemoryKind<M::MemoryKind>,
) -> InterpResult<'tcx> {
let (alloc_id, offset, prov) = self.ptr_get_alloc_id(ptr, 0)?;
trace!("deallocating: {alloc_id:?}");
if offset.bytes() != 0 {
throw_ub_custom!(
fluent::const_eval_realloc_or_alloc_with_offset,
ptr = format!("{ptr:?}"),
kind = "dealloc",
);
}
let Some((alloc_kind, mut alloc)) = self.memory.alloc_map.remove(&alloc_id) else {
// Deallocating global memory -- always an error
return Err(match self.tcx.try_get_global_alloc(alloc_id) {
Some(GlobalAlloc::Function { .. }) => {
err_ub_custom!(
fluent::const_eval_invalid_dealloc,
alloc_id = alloc_id,
kind = "fn",
)
}
Some(GlobalAlloc::VTable(..)) => {
err_ub_custom!(
fluent::const_eval_invalid_dealloc,
alloc_id = alloc_id,
kind = "vtable",
)
}
Some(GlobalAlloc::Static(..) | GlobalAlloc::Memory(..)) => {
err_ub_custom!(
fluent::const_eval_invalid_dealloc,
alloc_id = alloc_id,
kind = "static_mem"
)
}
None => err_ub!(PointerUseAfterFree(alloc_id, CheckInAllocMsg::MemoryAccessTest)),
}
.into());
};
if alloc.mutability.is_not() {
throw_ub_custom!(fluent::const_eval_dealloc_immutable, alloc = alloc_id,);
}
if alloc_kind != kind {
throw_ub_custom!(
fluent::const_eval_dealloc_kind_mismatch,
alloc = alloc_id,
alloc_kind = format!("{alloc_kind}"),
kind = format!("{kind}"),
);
}
if let Some((size, align)) = old_size_and_align {
if size != alloc.size() || align != alloc.align {
throw_ub_custom!(
fluent::const_eval_dealloc_incorrect_layout,
alloc = alloc_id,
size = alloc.size().bytes(),
align = alloc.align.bytes(),
size_found = size.bytes(),
align_found = align.bytes(),
)
}
}
// Let the machine take some extra action
let size = alloc.size();
M::before_memory_deallocation(
self.tcx,
&mut self.machine,
&mut alloc.extra,
(alloc_id, prov),
size,
alloc.align,
kind,
)?;
// Don't forget to remember size and align of this now-dead allocation
let old = self.memory.dead_alloc_map.insert(alloc_id, (size, alloc.align));
if old.is_some() {
bug!("Nothing can be deallocated twice");
}
Ok(())
}
/// Internal helper function to determine the allocation and offset of a pointer (if any).
#[inline(always)]
fn get_ptr_access(
&self,
ptr: Pointer<Option<M::Provenance>>,
size: Size,
) -> InterpResult<'tcx, Option<(AllocId, Size, M::ProvenanceExtra)>> {
let size = i64::try_from(size.bytes()).unwrap(); // it would be an error to even ask for more than isize::MAX bytes
self.check_and_deref_ptr(
ptr,
size,
CheckInAllocMsg::MemoryAccessTest,
|alloc_id, offset, prov| {
let (size, align) = self
.get_live_alloc_size_and_align(alloc_id, CheckInAllocMsg::MemoryAccessTest)?;
Ok((size, align, (alloc_id, offset, prov)))
},
)
}
/// Check if the given pointer points to live memory of the given `size`.
/// The caller can control the error message for the out-of-bounds case.
#[inline(always)]
pub fn check_ptr_access(
&self,
ptr: Pointer<Option<M::Provenance>>,
size: Size,
msg: CheckInAllocMsg,
) -> InterpResult<'tcx> {
let size = i64::try_from(size.bytes()).unwrap(); // it would be an error to even ask for more than isize::MAX bytes
self.check_and_deref_ptr(ptr, size, msg, |alloc_id, _, _| {
let (size, align) = self.get_live_alloc_size_and_align(alloc_id, msg)?;
Ok((size, align, ()))
})?;
Ok(())
}
/// Check whether the given pointer points to live memory for a signed amount of bytes.
/// A negative amounts means that the given range of memory to the left of the pointer
/// needs to be dereferenceable.
pub fn check_ptr_access_signed(
&self,
ptr: Pointer<Option<M::Provenance>>,
size: i64,
msg: CheckInAllocMsg,
) -> InterpResult<'tcx> {
self.check_and_deref_ptr(ptr, size, msg, |alloc_id, _, _| {
let (size, align) = self.get_live_alloc_size_and_align(alloc_id, msg)?;
Ok((size, align, ()))
})?;
Ok(())
}
/// Low-level helper function to check if a ptr is in-bounds and potentially return a reference
/// to the allocation it points to. Supports both shared and mutable references, as the actual
/// checking is offloaded to a helper closure. Supports signed sizes for checks "to the left" of
/// a pointer.
///
/// `alloc_size` will only get called for non-zero-sized accesses.
///
/// Returns `None` if and only if the size is 0.
fn check_and_deref_ptr<T>(
&self,
ptr: Pointer<Option<M::Provenance>>,
size: i64,
msg: CheckInAllocMsg,
alloc_size: impl FnOnce(
AllocId,
Size,
M::ProvenanceExtra,
) -> InterpResult<'tcx, (Size, Align, T)>,
) -> InterpResult<'tcx, Option<T>> {
// Everything is okay with size 0.
if size == 0 {
return Ok(None);
}
Ok(match self.ptr_try_get_alloc_id(ptr, size) {
Err(addr) => {
// We couldn't get a proper allocation.
throw_ub!(DanglingIntPointer { addr, inbounds_size: size, msg });
}
Ok((alloc_id, offset, prov)) => {
let (alloc_size, _alloc_align, ret_val) = alloc_size(alloc_id, offset, prov)?;
let offset = offset.bytes();
// Compute absolute begin and end of the range.
let (begin, end) = if size >= 0 {
(Some(offset), offset.checked_add(size as u64))
} else {
(offset.checked_sub(size.unsigned_abs()), Some(offset))
};
// Ensure both are within bounds.
let in_bounds = begin.is_some() && end.is_some_and(|e| e <= alloc_size.bytes());
if !in_bounds {
throw_ub!(PointerOutOfBounds {
alloc_id,
alloc_size,
ptr_offset: self.sign_extend_to_target_isize(offset),
inbounds_size: size,
msg,
})
}
Some(ret_val)
}
})
}
pub(super) fn check_misalign(
&self,
misaligned: Option<Misalignment>,
msg: CheckAlignMsg,
) -> InterpResult<'tcx> {
if let Some(misaligned) = misaligned {
throw_ub!(AlignmentCheckFailed(misaligned, msg))
}
Ok(())
}
pub(super) fn is_ptr_misaligned(
&self,
ptr: Pointer<Option<M::Provenance>>,
align: Align,
) -> Option<Misalignment> {
if !M::enforce_alignment(self) || align.bytes() == 1 {
return None;
}
#[inline]
fn is_offset_misaligned(offset: u64, align: Align) -> Option<Misalignment> {
if offset % align.bytes() == 0 {
None
} else {
// The biggest power of two through which `offset` is divisible.
let offset_pow2 = 1 << offset.trailing_zeros();
Some(Misalignment { has: Align::from_bytes(offset_pow2).unwrap(), required: align })
}
}
match self.ptr_try_get_alloc_id(ptr, 0) {
Err(addr) => is_offset_misaligned(addr, align),
Ok((alloc_id, offset, _prov)) => {
let (_size, alloc_align, kind) = self.get_alloc_info(alloc_id);
if let Some(misalign) =
M::alignment_check(self, alloc_id, alloc_align, kind, offset, align)
{
Some(misalign)
} else if M::Provenance::OFFSET_IS_ADDR {
is_offset_misaligned(ptr.addr().bytes(), align)
} else {
// Check allocation alignment and offset alignment.
if alloc_align.bytes() < align.bytes() {
Some(Misalignment { has: alloc_align, required: align })
} else {
is_offset_misaligned(offset.bytes(), align)
}
}
}
}
}
/// Checks a pointer for misalignment.
///
/// The error assumes this is checking the pointer used directly for an access.
pub fn check_ptr_align(
&self,
ptr: Pointer<Option<M::Provenance>>,
align: Align,
) -> InterpResult<'tcx> {
self.check_misalign(self.is_ptr_misaligned(ptr, align), CheckAlignMsg::AccessedPtr)
}
}
impl<'tcx, M: Machine<'tcx>> InterpCx<'tcx, M> {
/// This function is used by Miri's provenance GC to remove unreachable entries from the dead_alloc_map.
pub fn remove_unreachable_allocs(&mut self, reachable_allocs: &FxHashSet<AllocId>) {
// Unlike all the other GC helpers where we check if an `AllocId` is found in the interpreter or
// is live, here all the IDs in the map are for dead allocations so we don't
// need to check for liveness.
#[allow(rustc::potential_query_instability)] // Only used from Miri, not queries.
self.memory.dead_alloc_map.retain(|id, _| reachable_allocs.contains(id));
}
}
/// Allocation accessors
impl<'tcx, M: Machine<'tcx>> InterpCx<'tcx, M> {
/// Helper function to obtain a global (tcx) allocation.
/// This attempts to return a reference to an existing allocation if
/// one can be found in `tcx`. That, however, is only possible if `tcx` and
/// this machine use the same pointer provenance, so it is indirected through
/// `M::adjust_allocation`.
fn get_global_alloc(
&self,
id: AllocId,
is_write: bool,
) -> InterpResult<'tcx, Cow<'tcx, Allocation<M::Provenance, M::AllocExtra, M::Bytes>>> {
let (alloc, def_id) = match self.tcx.try_get_global_alloc(id) {
Some(GlobalAlloc::Memory(mem)) => {
// Memory of a constant or promoted or anonymous memory referenced by a static.
(mem, None)
}
Some(GlobalAlloc::Function { .. }) => throw_ub!(DerefFunctionPointer(id)),
Some(GlobalAlloc::VTable(..)) => throw_ub!(DerefVTablePointer(id)),
None => throw_ub!(PointerUseAfterFree(id, CheckInAllocMsg::MemoryAccessTest)),
Some(GlobalAlloc::Static(def_id)) => {
assert!(self.tcx.is_static(def_id));
// Thread-local statics do not have a constant address. They *must* be accessed via
// `ThreadLocalRef`; we can never have a pointer to them as a regular constant value.
assert!(!self.tcx.is_thread_local_static(def_id));
// Notice that every static has two `AllocId` that will resolve to the same
// thing here: one maps to `GlobalAlloc::Static`, this is the "lazy" ID,
// and the other one is maps to `GlobalAlloc::Memory`, this is returned by
// `eval_static_initializer` and it is the "resolved" ID.
// The resolved ID is never used by the interpreted program, it is hidden.
// This is relied upon for soundness of const-patterns; a pointer to the resolved
// ID would "sidestep" the checks that make sure consts do not point to statics!
// The `GlobalAlloc::Memory` branch here is still reachable though; when a static
// contains a reference to memory that was created during its evaluation (i.e., not
// to another static), those inner references only exist in "resolved" form.
if self.tcx.is_foreign_item(def_id) {
// This is unreachable in Miri, but can happen in CTFE where we actually *do* support
// referencing arbitrary (declared) extern statics.
throw_unsup!(ExternStatic(def_id));
}
// We don't give a span -- statics don't need that, they cannot be generic or associated.
let val = self.ctfe_query(|tcx| tcx.eval_static_initializer(def_id))?;
(val, Some(def_id))
}
};
M::before_access_global(self.tcx, &self.machine, id, alloc, def_id, is_write)?;
// We got tcx memory. Let the machine initialize its "extra" stuff.
M::adjust_global_allocation(
self,
id, // always use the ID we got as input, not the "hidden" one.
alloc.inner(),
)
}
/// Gives raw access to the `Allocation`, without bounds or alignment checks.
/// The caller is responsible for calling the access hooks!
///
/// You almost certainly want to use `get_ptr_alloc`/`get_ptr_alloc_mut` instead.
fn get_alloc_raw(
&self,
id: AllocId,
) -> InterpResult<'tcx, &Allocation<M::Provenance, M::AllocExtra, M::Bytes>> {
// The error type of the inner closure here is somewhat funny. We have two
// ways of "erroring": An actual error, or because we got a reference from
// `get_global_alloc` that we can actually use directly without inserting anything anywhere.
// So the error type is `InterpResult<'tcx, &Allocation<M::Provenance>>`.
let a = self.memory.alloc_map.get_or(id, || {
let alloc = self.get_global_alloc(id, /*is_write*/ false).map_err(Err)?;
match alloc {
Cow::Borrowed(alloc) => {
// We got a ref, cheaply return that as an "error" so that the
// map does not get mutated.
Err(Ok(alloc))
}
Cow::Owned(alloc) => {
// Need to put it into the map and return a ref to that
let kind = M::GLOBAL_KIND.expect(
"I got a global allocation that I have to copy but the machine does \
not expect that to happen",
);
Ok((MemoryKind::Machine(kind), alloc))
}
}
});
// Now unpack that funny error type
match a {
Ok(a) => Ok(&a.1),
Err(a) => a,
}
}
/// Gives raw, immutable access to the `Allocation` address, without bounds or alignment checks.
/// The caller is responsible for calling the access hooks!
pub fn get_alloc_bytes_unchecked_raw(&self, id: AllocId) -> InterpResult<'tcx, *const u8> {
let alloc = self.get_alloc_raw(id)?;
Ok(alloc.get_bytes_unchecked_raw())
}
/// Bounds-checked *but not align-checked* allocation access.
pub fn get_ptr_alloc<'a>(
&'a self,
ptr: Pointer<Option<M::Provenance>>,
size: Size,
) -> InterpResult<'tcx, Option<AllocRef<'a, 'tcx, M::Provenance, M::AllocExtra, M::Bytes>>>
{
let size_i64 = i64::try_from(size.bytes()).unwrap(); // it would be an error to even ask for more than isize::MAX bytes
let ptr_and_alloc = self.check_and_deref_ptr(
ptr,
size_i64,
CheckInAllocMsg::MemoryAccessTest,
|alloc_id, offset, prov| {
let alloc = self.get_alloc_raw(alloc_id)?;
Ok((alloc.size(), alloc.align, (alloc_id, offset, prov, alloc)))
},
)?;
// We want to call the hook on *all* accesses that involve an AllocId, including zero-sized
// accesses. That means we cannot rely on the closure above or the `Some` branch below. We
// do this after `check_and_deref_ptr` to ensure some basic sanity has already been checked.
if !self.memory.validation_in_progress.get() {
if let Ok((alloc_id, ..)) = self.ptr_try_get_alloc_id(ptr, size_i64) {
M::before_alloc_read(self, alloc_id)?;
}
}
if let Some((alloc_id, offset, prov, alloc)) = ptr_and_alloc {
let range = alloc_range(offset, size);
if !self.memory.validation_in_progress.get() {
M::before_memory_read(
self.tcx,
&self.machine,
&alloc.extra,
(alloc_id, prov),
range,
)?;
}
Ok(Some(AllocRef { alloc, range, tcx: *self.tcx, alloc_id }))
} else {
Ok(None)
}
}
/// Return the `extra` field of the given allocation.
pub fn get_alloc_extra<'a>(&'a self, id: AllocId) -> InterpResult<'tcx, &'a M::AllocExtra> {
Ok(&self.get_alloc_raw(id)?.extra)
}
/// Return the `mutability` field of the given allocation.
pub fn get_alloc_mutability<'a>(&'a self, id: AllocId) -> InterpResult<'tcx, Mutability> {
Ok(self.get_alloc_raw(id)?.mutability)
}
/// Gives raw mutable access to the `Allocation`, without bounds or alignment checks.
/// The caller is responsible for calling the access hooks!
///
/// Also returns a ptr to `self.extra` so that the caller can use it in parallel with the
/// allocation.
fn get_alloc_raw_mut(
&mut self,
id: AllocId,
) -> InterpResult<'tcx, (&mut Allocation<M::Provenance, M::AllocExtra, M::Bytes>, &mut M)> {
// We have "NLL problem case #3" here, which cannot be worked around without loss of
// efficiency even for the common case where the key is in the map.
// <https://rust-lang.github.io/rfcs/2094-nll.html#problem-case-3-conditional-control-flow-across-functions>
// (Cannot use `get_mut_or` since `get_global_alloc` needs `&self`.)
if self.memory.alloc_map.get_mut(id).is_none() {
// Slow path.
// Allocation not found locally, go look global.
let alloc = self.get_global_alloc(id, /*is_write*/ true)?;
let kind = M::GLOBAL_KIND.expect(
"I got a global allocation that I have to copy but the machine does \
not expect that to happen",
);
self.memory.alloc_map.insert(id, (MemoryKind::Machine(kind), alloc.into_owned()));
}
let (_kind, alloc) = self.memory.alloc_map.get_mut(id).unwrap();
if alloc.mutability.is_not() {
throw_ub!(WriteToReadOnly(id))
}
Ok((alloc, &mut self.machine))
}
/// Gives raw, mutable access to the `Allocation` address, without bounds or alignment checks.
/// The caller is responsible for calling the access hooks!
pub fn get_alloc_bytes_unchecked_raw_mut(
&mut self,
id: AllocId,
) -> InterpResult<'tcx, *mut u8> {
let alloc = self.get_alloc_raw_mut(id)?.0;
Ok(alloc.get_bytes_unchecked_raw_mut())
}
/// Bounds-checked *but not align-checked* allocation access.
pub fn get_ptr_alloc_mut<'a>(
&'a mut self,
ptr: Pointer<Option<M::Provenance>>,
size: Size,
) -> InterpResult<'tcx, Option<AllocRefMut<'a, 'tcx, M::Provenance, M::AllocExtra, M::Bytes>>>
{
let parts = self.get_ptr_access(ptr, size)?;
if let Some((alloc_id, offset, prov)) = parts {
let tcx = self.tcx;
// FIXME: can we somehow avoid looking up the allocation twice here?
// We cannot call `get_raw_mut` inside `check_and_deref_ptr` as that would duplicate `&mut self`.
let (alloc, machine) = self.get_alloc_raw_mut(alloc_id)?;
let range = alloc_range(offset, size);
M::before_memory_write(tcx, machine, &mut alloc.extra, (alloc_id, prov), range)?;
Ok(Some(AllocRefMut { alloc, range, tcx: *tcx, alloc_id }))
} else {
Ok(None)
}
}
/// Return the `extra` field of the given allocation.
pub fn get_alloc_extra_mut<'a>(
&'a mut self,
id: AllocId,
) -> InterpResult<'tcx, (&'a mut M::AllocExtra, &'a mut M)> {
let (alloc, machine) = self.get_alloc_raw_mut(id)?;
Ok((&mut alloc.extra, machine))
}
/// Check whether an allocation is live. This is faster than calling
/// [`InterpCx::get_alloc_info`] if all you need to check is whether the kind is
/// [`AllocKind::Dead`] because it doesn't have to look up the type and layout of statics.
pub fn is_alloc_live(&self, id: AllocId) -> bool {
self.tcx.try_get_global_alloc(id).is_some()
|| self.memory.alloc_map.contains_key_ref(&id)
|| self.memory.extra_fn_ptr_map.contains_key(&id)
}
/// Obtain the size and alignment of an allocation, even if that allocation has
/// been deallocated.
pub fn get_alloc_info(&self, id: AllocId) -> (Size, Align, AllocKind) {
// # Regular allocations
// Don't use `self.get_raw` here as that will
// a) cause cycles in case `id` refers to a static
// b) duplicate a global's allocation in miri
if let Some((_, alloc)) = self.memory.alloc_map.get(id) {
return (alloc.size(), alloc.align, AllocKind::LiveData);
}
// # Function pointers
// (both global from `alloc_map` and local from `extra_fn_ptr_map`)
if self.get_fn_alloc(id).is_some() {
return (Size::ZERO, Align::ONE, AllocKind::Function);
}
// # Statics
// Can't do this in the match argument, we may get cycle errors since the lock would
// be held throughout the match.
match self.tcx.try_get_global_alloc(id) {
Some(GlobalAlloc::Static(def_id)) => {
// Thread-local statics do not have a constant address. They *must* be accessed via
// `ThreadLocalRef`; we can never have a pointer to them as a regular constant value.
assert!(!self.tcx.is_thread_local_static(def_id));
let DefKind::Static { nested, .. } = self.tcx.def_kind(def_id) else {
bug!("GlobalAlloc::Static is not a static")
};
let (size, align) = if nested {
// Nested anonymous statics are untyped, so let's get their
// size and alignment from the allocaiton itself. This always
// succeeds, as the query is fed at DefId creation time, so no
// evaluation actually occurs.
let alloc = self.tcx.eval_static_initializer(def_id).unwrap();
(alloc.0.size(), alloc.0.align)
} else {
// Use size and align of the type for everything else. We need
// to do that to
// * avoid cycle errors in case of self-referential statics,
// * be able to get information on extern statics.
let ty = self
.tcx
.type_of(def_id)
.no_bound_vars()
.expect("statics should not have generic parameters");
let layout = self.tcx.layout_of(ParamEnv::empty().and(ty)).unwrap();
assert!(layout.is_sized());
(layout.size, layout.align.abi)
};
(size, align, AllocKind::LiveData)
}
Some(GlobalAlloc::Memory(alloc)) => {
// Need to duplicate the logic here, because the global allocations have
// different associated types than the interpreter-local ones.
let alloc = alloc.inner();
(alloc.size(), alloc.align, AllocKind::LiveData)
}
Some(GlobalAlloc::Function { .. }) => {
bug!("We already checked function pointers above")
}
Some(GlobalAlloc::VTable(..)) => {
// No data to be accessed here. But vtables are pointer-aligned.
return (Size::ZERO, self.tcx.data_layout.pointer_align.abi, AllocKind::VTable);
}
// The rest must be dead.
None => {
// Deallocated pointers are allowed, we should be able to find
// them in the map.
let (size, align) = *self
.memory
.dead_alloc_map
.get(&id)
.expect("deallocated pointers should all be recorded in `dead_alloc_map`");
(size, align, AllocKind::Dead)
}
}
}
/// Obtain the size and alignment of a *live* allocation.
fn get_live_alloc_size_and_align(
&self,
id: AllocId,
msg: CheckInAllocMsg,
) -> InterpResult<'tcx, (Size, Align)> {
let (size, align, kind) = self.get_alloc_info(id);
if matches!(kind, AllocKind::Dead) {
throw_ub!(PointerUseAfterFree(id, msg))
}
Ok((size, align))
}
fn get_fn_alloc(&self, id: AllocId) -> Option<FnVal<'tcx, M::ExtraFnVal>> {
if let Some(extra) = self.memory.extra_fn_ptr_map.get(&id) {
Some(FnVal::Other(*extra))
} else {
match self.tcx.try_get_global_alloc(id) {
Some(GlobalAlloc::Function { instance, .. }) => Some(FnVal::Instance(instance)),
_ => None,
}
}
}
pub fn get_ptr_fn(
&self,
ptr: Pointer<Option<M::Provenance>>,
) -> InterpResult<'tcx, FnVal<'tcx, M::ExtraFnVal>> {
trace!("get_ptr_fn({:?})", ptr);
let (alloc_id, offset, _prov) = self.ptr_get_alloc_id(ptr, 0)?;
if offset.bytes() != 0 {
throw_ub!(InvalidFunctionPointer(Pointer::new(alloc_id, offset)))
}
self.get_fn_alloc(alloc_id)
.ok_or_else(|| err_ub!(InvalidFunctionPointer(Pointer::new(alloc_id, offset))).into())
}
/// Get the dynamic type of the given vtable pointer.
/// If `expected_trait` is `Some`, it must be a vtable for the given trait.
pub fn get_ptr_vtable_ty(
&self,
ptr: Pointer<Option<M::Provenance>>,
expected_trait: Option<&'tcx ty::List<ty::PolyExistentialPredicate<'tcx>>>,
) -> InterpResult<'tcx, Ty<'tcx>> {
trace!("get_ptr_vtable({:?})", ptr);
let (alloc_id, offset, _tag) = self.ptr_get_alloc_id(ptr, 0)?;
if offset.bytes() != 0 {
throw_ub!(InvalidVTablePointer(Pointer::new(alloc_id, offset)))
}
let Some(GlobalAlloc::VTable(ty, vtable_trait)) = self.tcx.try_get_global_alloc(alloc_id)
else {
throw_ub!(InvalidVTablePointer(Pointer::new(alloc_id, offset)))
};
if let Some(expected_trait) = expected_trait {
self.check_vtable_for_type(vtable_trait, expected_trait)?;
}
Ok(ty)
}
pub fn alloc_mark_immutable(&mut self, id: AllocId) -> InterpResult<'tcx> {
self.get_alloc_raw_mut(id)?.0.mutability = Mutability::Not;
Ok(())
}
/// Create a lazy debug printer that prints the given allocation and all allocations it points
/// to, recursively.
#[must_use]
pub fn dump_alloc<'a>(&'a self, id: AllocId) -> DumpAllocs<'a, 'tcx, M> {
self.dump_allocs(vec![id])
}
/// Create a lazy debug printer for a list of allocations and all allocations they point to,
/// recursively.
#[must_use]
pub fn dump_allocs<'a>(&'a self, mut allocs: Vec<AllocId>) -> DumpAllocs<'a, 'tcx, M> {
allocs.sort();
allocs.dedup();
DumpAllocs { ecx: self, allocs }
}
/// Print the allocation's bytes, without any nested allocations.
pub fn print_alloc_bytes_for_diagnostics(&self, id: AllocId) -> String {
// Using the "raw" access to avoid the `before_alloc_read` hook, we specifically
// want to be able to read all memory for diagnostics, even if that is cyclic.
let alloc = self.get_alloc_raw(id).unwrap();
let mut bytes = String::new();
if alloc.size() != Size::ZERO {
bytes = "\n".into();
// FIXME(translation) there might be pieces that are translatable.
rustc_middle::mir::pretty::write_allocation_bytes(*self.tcx, alloc, &mut bytes, " ")
.unwrap();
}
bytes
}
/// Find leaked allocations. Allocations reachable from `static_roots` or a `Global` allocation
/// are not considered leaked, as well as leaks whose kind's `may_leak()` returns true.
pub fn find_leaked_allocations(
&self,
static_roots: &[AllocId],
) -> Vec<(AllocId, MemoryKind<M::MemoryKind>, Allocation<M::Provenance, M::AllocExtra, M::Bytes>)>
{
// Collect the set of allocations that are *reachable* from `Global` allocations.
let reachable = {
let mut reachable = FxHashSet::default();
let global_kind = M::GLOBAL_KIND.map(MemoryKind::Machine);
let mut todo: Vec<_> =
self.memory.alloc_map.filter_map_collect(move |&id, &(kind, _)| {
if Some(kind) == global_kind { Some(id) } else { None }
});
todo.extend(static_roots);
while let Some(id) = todo.pop() {
if reachable.insert(id) {
// This is a new allocation, add the allocation it points to `todo`.
if let Some((_, alloc)) = self.memory.alloc_map.get(id) {
todo.extend(
alloc.provenance().provenances().filter_map(|prov| prov.get_alloc_id()),
);
}
}
}
reachable
};
// All allocations that are *not* `reachable` and *not* `may_leak` are considered leaking.
self.memory.alloc_map.filter_map_collect(|id, (kind, alloc)| {
if kind.may_leak() || reachable.contains(id) {
None
} else {
Some((*id, *kind, alloc.clone()))
}
})
}
/// Runs the closure in "validation" mode, which means the machine's memory read hooks will be
/// suppressed. Needless to say, this must only be set with great care! Cannot be nested.
///
/// We do this so Miri's allocation access tracking does not show the validation
/// reads as spurious accesses.
pub fn run_for_validation<R>(&self, f: impl FnOnce() -> R) -> R {
// This deliberately uses `==` on `bool` to follow the pattern
// `assert!(val.replace(new) == old)`.
assert!(
self.memory.validation_in_progress.replace(true) == false,
"`validation_in_progress` was already set"
);
let res = f();
assert!(
self.memory.validation_in_progress.replace(false) == true,
"`validation_in_progress` was unset by someone else"
);
res
}
}
#[doc(hidden)]
/// There's no way to use this directly, it's just a helper struct for the `dump_alloc(s)` methods.
pub struct DumpAllocs<'a, 'tcx, M: Machine<'tcx>> {
ecx: &'a InterpCx<'tcx, M>,
allocs: Vec<AllocId>,
}
impl<'a, 'tcx, M: Machine<'tcx>> std::fmt::Debug for DumpAllocs<'a, 'tcx, M> {
fn fmt(&self, fmt: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
// Cannot be a closure because it is generic in `Prov`, `Extra`.
fn write_allocation_track_relocs<'tcx, Prov: Provenance, Extra, Bytes: AllocBytes>(
fmt: &mut std::fmt::Formatter<'_>,
tcx: TyCtxt<'tcx>,
allocs_to_print: &mut VecDeque<AllocId>,
alloc: &Allocation<Prov, Extra, Bytes>,
) -> std::fmt::Result {
for alloc_id in alloc.provenance().provenances().filter_map(|prov| prov.get_alloc_id())
{
allocs_to_print.push_back(alloc_id);
}
write!(fmt, "{}", display_allocation(tcx, alloc))
}
let mut allocs_to_print: VecDeque<_> = self.allocs.iter().copied().collect();
// `allocs_printed` contains all allocations that we have already printed.
let mut allocs_printed = FxHashSet::default();
while let Some(id) = allocs_to_print.pop_front() {
if !allocs_printed.insert(id) {
// Already printed, so skip this.
continue;
}
write!(fmt, "{id:?}")?;
match self.ecx.memory.alloc_map.get(id) {
Some((kind, alloc)) => {
// normal alloc
write!(fmt, " ({kind}, ")?;
write_allocation_track_relocs(
&mut *fmt,
*self.ecx.tcx,
&mut allocs_to_print,
alloc,
)?;
}
None => {
// global alloc
match self.ecx.tcx.try_get_global_alloc(id) {
Some(GlobalAlloc::Memory(alloc)) => {
write!(fmt, " (unchanged global, ")?;
write_allocation_track_relocs(
&mut *fmt,
*self.ecx.tcx,
&mut allocs_to_print,
alloc.inner(),
)?;
}
Some(GlobalAlloc::Function { instance, .. }) => {
write!(fmt, " (fn: {instance})")?;
}
Some(GlobalAlloc::VTable(ty, Some(trait_ref))) => {
write!(fmt, " (vtable: impl {trait_ref} for {ty})")?;
}
Some(GlobalAlloc::VTable(ty, None)) => {
write!(fmt, " (vtable: impl <auto trait> for {ty})")?;
}
Some(GlobalAlloc::Static(did)) => {
write!(fmt, " (static: {})", self.ecx.tcx.def_path_str(did))?;
}
None => {
write!(fmt, " (deallocated)")?;
}
}
}
}
writeln!(fmt)?;
}
Ok(())
}
}
/// Reading and writing.
impl<'tcx, 'a, Prov: Provenance, Extra, Bytes: AllocBytes>
AllocRefMut<'a, 'tcx, Prov, Extra, Bytes>
{
/// `range` is relative to this allocation reference, not the base of the allocation.
pub fn write_scalar(&mut self, range: AllocRange, val: Scalar<Prov>) -> InterpResult<'tcx> {
let range = self.range.subrange(range);
debug!("write_scalar at {:?}{range:?}: {val:?}", self.alloc_id);
Ok(self
.alloc
.write_scalar(&self.tcx, range, val)
.map_err(|e| e.to_interp_error(self.alloc_id))?)
}
/// `offset` is relative to this allocation reference, not the base of the allocation.
pub fn write_ptr_sized(&mut self, offset: Size, val: Scalar<Prov>) -> InterpResult<'tcx> {
self.write_scalar(alloc_range(offset, self.tcx.data_layout().pointer_size), val)
}
/// Mark the entire referenced range as uninitialized
pub fn write_uninit(&mut self) -> InterpResult<'tcx> {
Ok(self
.alloc
.write_uninit(&self.tcx, self.range)
.map_err(|e| e.to_interp_error(self.alloc_id))?)
}
}
impl<'tcx, 'a, Prov: Provenance, Extra, Bytes: AllocBytes> AllocRef<'a, 'tcx, Prov, Extra, Bytes> {
/// `range` is relative to this allocation reference, not the base of the allocation.
pub fn read_scalar(
&self,
range: AllocRange,
read_provenance: bool,
) -> InterpResult<'tcx, Scalar<Prov>> {
let range = self.range.subrange(range);
let res = self
.alloc
.read_scalar(&self.tcx, range, read_provenance)
.map_err(|e| e.to_interp_error(self.alloc_id))?;
debug!("read_scalar at {:?}{range:?}: {res:?}", self.alloc_id);
Ok(res)
}
/// `range` is relative to this allocation reference, not the base of the allocation.
pub fn read_integer(&self, range: AllocRange) -> InterpResult<'tcx, Scalar<Prov>> {
self.read_scalar(range, /*read_provenance*/ false)
}
/// `offset` is relative to this allocation reference, not the base of the allocation.
pub fn read_pointer(&self, offset: Size) -> InterpResult<'tcx, Scalar<Prov>> {
self.read_scalar(
alloc_range(offset, self.tcx.data_layout().pointer_size),
/*read_provenance*/ true,
)
}
/// `range` is relative to this allocation reference, not the base of the allocation.
pub fn get_bytes_strip_provenance<'b>(&'b self) -> InterpResult<'tcx, &'a [u8]> {
Ok(self
.alloc
.get_bytes_strip_provenance(&self.tcx, self.range)
.map_err(|e| e.to_interp_error(self.alloc_id))?)
}
/// Returns whether the allocation has provenance anywhere in the range of the `AllocRef`.
pub fn has_provenance(&self) -> bool {
!self.alloc.provenance().range_empty(self.range, &self.tcx)
}
}
impl<'tcx, M: Machine<'tcx>> InterpCx<'tcx, M> {
/// Reads the given number of bytes from memory, and strips their provenance if possible.
/// Returns them as a slice.
///
/// Performs appropriate bounds checks.
pub fn read_bytes_ptr_strip_provenance(
&self,
ptr: Pointer<Option<M::Provenance>>,
size: Size,
) -> InterpResult<'tcx, &[u8]> {
let Some(alloc_ref) = self.get_ptr_alloc(ptr, size)? else {
// zero-sized access
return Ok(&[]);
};
// Side-step AllocRef and directly access the underlying bytes more efficiently.
// (We are staying inside the bounds here so all is good.)
Ok(alloc_ref
.alloc
.get_bytes_strip_provenance(&alloc_ref.tcx, alloc_ref.range)
.map_err(|e| e.to_interp_error(alloc_ref.alloc_id))?)
}
/// Writes the given stream of bytes into memory.
///
/// Performs appropriate bounds checks.
pub fn write_bytes_ptr(
&mut self,
ptr: Pointer<Option<M::Provenance>>,
src: impl IntoIterator<Item = u8>,
) -> InterpResult<'tcx> {
let mut src = src.into_iter();
let (lower, upper) = src.size_hint();
let len = upper.expect("can only write bounded iterators");
assert_eq!(lower, len, "can only write iterators with a precise length");
let size = Size::from_bytes(len);
let Some(alloc_ref) = self.get_ptr_alloc_mut(ptr, size)? else {
// zero-sized access
assert_matches!(src.next(), None, "iterator said it was empty but returned an element");
return Ok(());
};
// Side-step AllocRef and directly access the underlying bytes more efficiently.
// (We are staying inside the bounds here and all bytes do get overwritten so all is good.)
let alloc_id = alloc_ref.alloc_id;
let bytes = alloc_ref
.alloc
.get_bytes_unchecked_for_overwrite(&alloc_ref.tcx, alloc_ref.range)
.map_err(move |e| e.to_interp_error(alloc_id))?;
// `zip` would stop when the first iterator ends; we want to definitely
// cover all of `bytes`.
for dest in bytes {
*dest = src.next().expect("iterator was shorter than it said it would be");
}
assert_matches!(src.next(), None, "iterator was longer than it said it would be");
Ok(())
}
pub fn mem_copy(
&mut self,
src: Pointer<Option<M::Provenance>>,
dest: Pointer<Option<M::Provenance>>,
size: Size,
nonoverlapping: bool,
) -> InterpResult<'tcx> {
self.mem_copy_repeatedly(src, dest, size, 1, nonoverlapping)
}
/// Performs `num_copies` many copies of `size` many bytes from `src` to `dest + i*size` (where
/// `i` is the index of the copy).
///
/// Either `nonoverlapping` must be true or `num_copies` must be 1; doing repeated copies that
/// may overlap is not supported.
pub fn mem_copy_repeatedly(
&mut self,
src: Pointer<Option<M::Provenance>>,
dest: Pointer<Option<M::Provenance>>,
size: Size,
num_copies: u64,
nonoverlapping: bool,
) -> InterpResult<'tcx> {
let tcx = self.tcx;
// We need to do our own bounds-checks.
let src_parts = self.get_ptr_access(src, size)?;
let dest_parts = self.get_ptr_access(dest, size * num_copies)?; // `Size` multiplication
// FIXME: we look up both allocations twice here, once before for the `check_ptr_access`
// and once below to get the underlying `&[mut] Allocation`.
// Source alloc preparations and access hooks.
let Some((src_alloc_id, src_offset, src_prov)) = src_parts else {
// Zero-sized *source*, that means dest is also zero-sized and we have nothing to do.
return Ok(());
};
let src_alloc = self.get_alloc_raw(src_alloc_id)?;
let src_range = alloc_range(src_offset, size);
assert!(!self.memory.validation_in_progress.get(), "we can't be copying during validation");
M::before_memory_read(
tcx,
&self.machine,
&src_alloc.extra,
(src_alloc_id, src_prov),
src_range,
)?;
// We need the `dest` ptr for the next operation, so we get it now.
// We already did the source checks and called the hooks so we are good to return early.
let Some((dest_alloc_id, dest_offset, dest_prov)) = dest_parts else {
// Zero-sized *destination*.
return Ok(());
};
// Prepare getting source provenance.
let src_bytes = src_alloc.get_bytes_unchecked(src_range).as_ptr(); // raw ptr, so we can also get a ptr to the destination allocation
// first copy the provenance to a temporary buffer, because
// `get_bytes_mut` will clear the provenance, which is correct,
// since we don't want to keep any provenance at the target.
// This will also error if copying partial provenance is not supported.
let provenance = src_alloc
.provenance()
.prepare_copy(src_range, dest_offset, num_copies, self)
.map_err(|e| e.to_interp_error(dest_alloc_id))?;
// Prepare a copy of the initialization mask.
let init = src_alloc.init_mask().prepare_copy(src_range);
// Destination alloc preparations and access hooks.
let (dest_alloc, extra) = self.get_alloc_raw_mut(dest_alloc_id)?;
let dest_range = alloc_range(dest_offset, size * num_copies);
M::before_memory_write(
tcx,
extra,
&mut dest_alloc.extra,
(dest_alloc_id, dest_prov),
dest_range,
)?;
// Yes we do overwrite all bytes in `dest_bytes`.
let dest_bytes = dest_alloc
.get_bytes_unchecked_for_overwrite_ptr(&tcx, dest_range)
.map_err(|e| e.to_interp_error(dest_alloc_id))?
.as_mut_ptr();
if init.no_bytes_init() {
// Fast path: If all bytes are `uninit` then there is nothing to copy. The target range
// is marked as uninitialized but we otherwise omit changing the byte representation which may
// be arbitrary for uninitialized bytes.
// This also avoids writing to the target bytes so that the backing allocation is never
// touched if the bytes stay uninitialized for the whole interpreter execution. On contemporary
// operating system this can avoid physically allocating the page.
dest_alloc
.write_uninit(&tcx, dest_range)
.map_err(|e| e.to_interp_error(dest_alloc_id))?;
// We can forget about the provenance, this is all not initialized anyway.
return Ok(());
}
// SAFE: The above indexing would have panicked if there weren't at least `size` bytes
// behind `src` and `dest`. Also, we use the overlapping-safe `ptr::copy` if `src` and
// `dest` could possibly overlap.
// The pointers above remain valid even if the `HashMap` table is moved around because they
// point into the `Vec` storing the bytes.
unsafe {
if src_alloc_id == dest_alloc_id {
if nonoverlapping {
// `Size` additions
if (src_offset <= dest_offset && src_offset + size > dest_offset)
|| (dest_offset <= src_offset && dest_offset + size > src_offset)
{
throw_ub_custom!(fluent::const_eval_copy_nonoverlapping_overlapping);
}
}
}
if num_copies > 1 {
assert!(nonoverlapping, "multi-copy only supported in non-overlapping mode");
}
let size_in_bytes = size.bytes_usize();
// For particularly large arrays (where this is perf-sensitive) it's common that
// we're writing a single byte repeatedly. So, optimize that case to a memset.
if size_in_bytes == 1 {
debug_assert!(num_copies >= 1); // we already handled the zero-sized cases above.
// SAFETY: `src_bytes` would be read from anyway by `copy` below (num_copies >= 1).
let value = *src_bytes;
dest_bytes.write_bytes(value, (size * num_copies).bytes_usize());
} else if src_alloc_id == dest_alloc_id {
let mut dest_ptr = dest_bytes;
for _ in 0..num_copies {
// Here we rely on `src` and `dest` being non-overlapping if there is more than
// one copy.
ptr::copy(src_bytes, dest_ptr, size_in_bytes);
dest_ptr = dest_ptr.add(size_in_bytes);
}
} else {
let mut dest_ptr = dest_bytes;
for _ in 0..num_copies {
ptr::copy_nonoverlapping(src_bytes, dest_ptr, size_in_bytes);
dest_ptr = dest_ptr.add(size_in_bytes);
}
}
}
// now fill in all the "init" data
dest_alloc.init_mask_apply_copy(
init,
alloc_range(dest_offset, size), // just a single copy (i.e., not full `dest_range`)
num_copies,
);
// copy the provenance to the destination
dest_alloc.provenance_apply_copy(provenance);
Ok(())
}
}
/// Machine pointer introspection.
impl<'tcx, M: Machine<'tcx>> InterpCx<'tcx, M> {
/// Test if this value might be null.
/// If the machine does not support ptr-to-int casts, this is conservative.
pub fn scalar_may_be_null(&self, scalar: Scalar<M::Provenance>) -> InterpResult<'tcx, bool> {
Ok(match scalar.try_to_scalar_int() {
Ok(int) => int.is_null(),
Err(_) => {
// Can only happen during CTFE.
let ptr = scalar.to_pointer(self)?;
match self.ptr_try_get_alloc_id(ptr, 0) {
Ok((alloc_id, offset, _)) => {
let (size, _align, _kind) = self.get_alloc_info(alloc_id);
// If the pointer is out-of-bounds, it may be null.
// Note that one-past-the-end (offset == size) is still inbounds, and never null.
offset > size
}
Err(_offset) => bug!("a non-int scalar is always a pointer"),
}
}
})
}
/// Turning a "maybe pointer" into a proper pointer (and some information
/// about where it points), or an absolute address.
///
/// `size` says how many bytes of memory are expected at that pointer. This is largely only used
/// for error messages; however, the *sign* of `size` can be used to disambiguate situations
/// where a wildcard pointer sits right in between two allocations.
/// It is almost always okay to just set the size to 0; this will be treated like a positive size
/// for handling wildcard pointers.
///
/// The result must be used immediately; it is not allowed to convert
/// the returned data back into a `Pointer` and store that in machine state.
/// (In fact that's not even possible since `M::ProvenanceExtra` is generic and
/// we don't have an operation to turn it back into `M::Provenance`.)
pub fn ptr_try_get_alloc_id(
&self,
ptr: Pointer<Option<M::Provenance>>,
size: i64,
) -> Result<(AllocId, Size, M::ProvenanceExtra), u64> {
match ptr.into_pointer_or_addr() {
Ok(ptr) => match M::ptr_get_alloc(self, ptr, size) {
Some((alloc_id, offset, extra)) => Ok((alloc_id, offset, extra)),
None => {
assert!(M::Provenance::OFFSET_IS_ADDR);
let (_, addr) = ptr.into_parts();
Err(addr.bytes())
}
},
Err(addr) => Err(addr.bytes()),
}
}
/// Turning a "maybe pointer" into a proper pointer (and some information about where it points).
///
/// `size` says how many bytes of memory are expected at that pointer. This is largely only used
/// for error messages; however, the *sign* of `size` can be used to disambiguate situations
/// where a wildcard pointer sits right in between two allocations.
/// It is almost always okay to just set the size to 0; this will be treated like a positive size
/// for handling wildcard pointers.
///
/// The result must be used immediately; it is not allowed to convert
/// the returned data back into a `Pointer` and store that in machine state.
/// (In fact that's not even possible since `M::ProvenanceExtra` is generic and
/// we don't have an operation to turn it back into `M::Provenance`.)
#[inline(always)]
pub fn ptr_get_alloc_id(
&self,
ptr: Pointer<Option<M::Provenance>>,
size: i64,
) -> InterpResult<'tcx, (AllocId, Size, M::ProvenanceExtra)> {
self.ptr_try_get_alloc_id(ptr, size).map_err(|offset| {
err_ub!(DanglingIntPointer {
addr: offset,
inbounds_size: size,
msg: CheckInAllocMsg::InboundsTest
})
.into()
})
}
}