rustc_mir_transform/dest_prop.rs
1//! Propagates assignment destinations backwards in the CFG to eliminate redundant assignments.
2//!
3//! # Motivation
4//!
5//! MIR building can insert a lot of redundant copies, and Rust code in general often tends to move
6//! values around a lot. The result is a lot of assignments of the form `dest = {move} src;` in MIR.
7//! MIR building for constants in particular tends to create additional locals that are only used
8//! inside a single block to shuffle a value around unnecessarily.
9//!
10//! LLVM by itself is not good enough at eliminating these redundant copies (eg. see
11//! <https://github.com/rust-lang/rust/issues/32966>), so this leaves some performance on the table
12//! that we can regain by implementing an optimization for removing these assign statements in rustc
13//! itself. When this optimization runs fast enough, it can also speed up the constant evaluation
14//! and code generation phases of rustc due to the reduced number of statements and locals.
15//!
16//! # The Optimization
17//!
18//! Conceptually, this optimization is "destination propagation". It is similar to the Named Return
19//! Value Optimization, or NRVO, known from the C++ world, except that it isn't limited to return
20//! values or the return place `_0`. On a very high level, independent of the actual implementation
21//! details, it does the following:
22//!
23//! 1) Identify `dest = src;` statements with values for `dest` and `src` whose storage can soundly
24//! be merged.
25//! 2) Replace all mentions of `src` with `dest` ("unifying" them and propagating the destination
26//! backwards).
27//! 3) Delete the `dest = src;` statement (by making it a `nop`).
28//!
29//! Step 1) is by far the hardest, so it is explained in more detail below.
30//!
31//! ## Soundness
32//!
33//! We have a pair of places `p` and `q`, whose memory we would like to merge. In order for this to
34//! be sound, we need to check a number of conditions:
35//!
36//! * `p` and `q` must both be *constant* - it does not make much sense to talk about merging them
37//! if they do not consistently refer to the same place in memory. This is satisfied if they do
38//! not contain any indirection through a pointer or any indexing projections.
39//!
40//! * `p` and `q` must have the **same type**. If we replace a local with a subtype or supertype,
41//! we may end up with a different vtable for that local. See the `subtyping-impacts-selection`
42//! tests for an example where that causes issues.
43//!
44//! * We need to make sure that the goal of "merging the memory" is actually structurally possible
45//! in MIR. For example, even if all the other conditions are satisfied, there is no way to
46//! "merge" `_5.foo` and `_6.bar`. For now, we ensure this by requiring that both `p` and `q` are
47//! locals with no further projections. Future iterations of this pass should improve on this.
48//!
49//! * Finally, we want `p` and `q` to use the same memory - however, we still need to make sure that
50//! each of them has enough "ownership" of that memory to continue "doing its job." More
51//! precisely, what we will check is that whenever the program performs a write to `p`, then it
52//! does not currently care about what the value in `q` is (and vice versa). We formalize the
53//! notion of "does not care what the value in `q` is" by checking the *liveness* of `q`.
54//!
55//! Because of the difficulty of computing liveness of places that have their address taken, we do
56//! not even attempt to do it. Any places that are in a local that has its address taken is
57//! excluded from the optimization.
58//!
59//! The first two conditions are simple structural requirements on the `Assign` statements that can
60//! be trivially checked. The third requirement however is more difficult and costly to check.
61//!
62//! ## Current implementation
63//!
64//! The current implementation relies on live range computation to check for conflicts. We only
65//! allow to merge locals that have disjoint live ranges. The live range are defined with
66//! half-statement granularity, so as to make all writes be live for at least a half statement.
67//!
68//! ## Future Improvements
69//!
70//! There are a number of ways in which this pass could be improved in the future:
71//!
72//! * Merging storage liveness ranges instead of removing storage statements completely. This may
73//! improve stack usage.
74//!
75//! * Allow merging locals into places with projections, eg `_5` into `_6.foo`.
76//!
77//! * Liveness analysis with more precision than whole locals at a time. The smaller benefit of this
78//! is that it would allow us to dest prop at "sub-local" levels in some cases. The bigger benefit
79//! of this is that such liveness analysis can report more accurate results about whole locals at
80//! a time. For example, consider:
81//!
82//! ```ignore (syntax-highlighting-only)
83//! _1 = u;
84//! // unrelated code
85//! _1.f1 = v;
86//! _2 = _1.f1;
87//! ```
88//!
89//! Because the current analysis only thinks in terms of locals, it does not have enough
90//! information to report that `_1` is dead in the "unrelated code" section.
91//!
92//! * Liveness analysis enabled by alias analysis. This would allow us to not just bail on locals
93//! that ever have their address taken. Of course that requires actually having alias analysis
94//! (and a model to build it on), so this might be a bit of a ways off.
95//!
96//! * Various perf improvements. There are a bunch of comments in here marked `PERF` with ideas for
97//! how to do things more efficiently. However, the complexity of the pass as a whole should be
98//! kept in mind.
99//!
100//! ## Previous Work
101//!
102//! A [previous attempt][attempt 1] at implementing an optimization like this turned out to be a
103//! significant regression in compiler performance. Fixing the regressions introduced a lot of
104//! undesirable complexity to the implementation.
105//!
106//! A [subsequent approach][attempt 2] tried to avoid the costly computation by limiting itself to
107//! acyclic CFGs, but still turned out to be far too costly to run due to suboptimal performance
108//! within individual basic blocks, requiring a walk across the entire block for every assignment
109//! found within the block. For the `tuple-stress` benchmark, which has 458745 statements in a
110//! single block, this proved to be far too costly.
111//!
112//! [Another approach after that][attempt 3] was much closer to correct, but had some soundness
113//! issues - it was failing to consider stores outside live ranges, and failed to uphold some of the
114//! requirements that MIR has for non-overlapping places within statements. However, it also had
115//! performance issues caused by `O(l² * s)` runtime, where `l` is the number of locals and `s` is
116//! the number of statements and terminators.
117//!
118//! Since the first attempt at this, the compiler has improved dramatically, and new analysis
119//! frameworks have been added that should make this approach viable without requiring a limited
120//! approach that only works for some classes of CFGs:
121//! - rustc now has a powerful dataflow analysis framework that can handle forwards and backwards
122//! analyses efficiently.
123//! - Layout optimizations for coroutines have been added to improve code generation for
124//! async/await, which are very similar in spirit to what this optimization does.
125//!
126//! [The next approach][attempt 4] computes a conflict matrix between locals by forbidding merging
127//! locals with competing writes or with one write while the other is live.
128//!
129//! ## Pre/Post Optimization
130//!
131//! It is recommended to run `SimplifyCfg` and then `SimplifyLocals` some time after this pass, as
132//! it replaces the eliminated assign statements with `nop`s and leaves unused locals behind.
133//!
134//! [liveness]: https://en.wikipedia.org/wiki/Live_variable_analysis
135//! [attempt 1]: https://github.com/rust-lang/rust/pull/47954
136//! [attempt 2]: https://github.com/rust-lang/rust/pull/71003
137//! [attempt 3]: https://github.com/rust-lang/rust/pull/72632
138//! [attempt 4]: https://github.com/rust-lang/rust/pull/96451
139
140use rustc_data_structures::union_find::UnionFind;
141use rustc_index::bit_set::DenseBitSet;
142use rustc_index::interval::SparseIntervalMatrix;
143use rustc_index::{IndexVec, newtype_index};
144use rustc_middle::mir::visit::{MutVisitor, PlaceContext, VisitPlacesWith, Visitor};
145use rustc_middle::mir::*;
146use rustc_middle::ty::TyCtxt;
147use rustc_mir_dataflow::impls::{DefUse, MaybeLiveLocals};
148use rustc_mir_dataflow::points::DenseLocationMap;
149use rustc_mir_dataflow::{Analysis, Results};
150use tracing::{debug, trace};
151
152pub(super) struct DestinationPropagation;
153
154impl<'tcx> crate::MirPass<'tcx> for DestinationPropagation {
155 fn is_enabled(&self, sess: &rustc_session::Session) -> bool {
156 sess.mir_opt_level() >= 2
157 }
158
159 #[tracing::instrument(level = "trace", skip(self, tcx, body))]
160 fn run_pass(&self, tcx: TyCtxt<'tcx>, body: &mut Body<'tcx>) {
161 let def_id = body.source.def_id();
162 trace!(?def_id);
163
164 let borrowed = rustc_mir_dataflow::impls::borrowed_locals(body);
165
166 let candidates = Candidates::find(body, &borrowed);
167 trace!(?candidates);
168 if candidates.c.is_empty() {
169 return;
170 }
171
172 let live = MaybeLiveLocals.iterate_to_fixpoint(tcx, body, Some("MaybeLiveLocals-DestProp"));
173
174 let points = DenseLocationMap::new(body);
175 let mut relevant = RelevantLocals::compute(&candidates, body.local_decls.len());
176 let mut live = save_as_intervals(&points, body, &relevant, live.results);
177
178 dest_prop_mir_dump(tcx, body, &points, &live, &relevant);
179
180 let mut merged_locals = DenseBitSet::new_empty(body.local_decls.len());
181
182 for (src, dst) in candidates.c.into_iter() {
183 trace!(?src, ?dst);
184
185 let Some(mut src) = relevant.find(src) else { continue };
186 let Some(mut dst) = relevant.find(dst) else { continue };
187 if src == dst {
188 continue;
189 }
190
191 let Some(src_live_ranges) = live.row(src) else { continue };
192 let Some(dst_live_ranges) = live.row(dst) else { continue };
193 trace!(?src, ?src_live_ranges);
194 trace!(?dst, ?dst_live_ranges);
195
196 if src_live_ranges.disjoint(dst_live_ranges) {
197 // We want to replace `src` by `dst`.
198 let mut orig_src = relevant.original[src];
199 let mut orig_dst = relevant.original[dst];
200
201 // The return place and function arguments are required and cannot be renamed.
202 // This check cannot be made during candidate collection, as we may want to
203 // unify the same non-required local with several required locals.
204 match (is_local_required(orig_src, body), is_local_required(orig_dst, body)) {
205 // Renaming `src` is ok.
206 (false, _) => {}
207 // Renaming `src` is wrong, but renaming `dst` is ok.
208 (true, false) => {
209 std::mem::swap(&mut src, &mut dst);
210 std::mem::swap(&mut orig_src, &mut orig_dst);
211 }
212 // Neither local can be renamed, so skip this case.
213 (true, true) => continue,
214 }
215
216 trace!(?src, ?dst, "merge");
217 merged_locals.insert(orig_src);
218 merged_locals.insert(orig_dst);
219
220 // Replace `src` by `dst`.
221 let head = relevant.union(src, dst);
222 live.union_rows(/* read */ src, /* write */ head);
223 live.union_rows(/* read */ dst, /* write */ head);
224 }
225 }
226 trace!(?merged_locals);
227 trace!(?relevant.renames);
228
229 if merged_locals.is_empty() {
230 return;
231 }
232
233 apply_merges(body, tcx, relevant, merged_locals);
234 }
235
236 fn is_required(&self) -> bool {
237 false
238 }
239}
240
241//////////////////////////////////////////////////////////
242// Merging
243//
244// Applies the actual optimization
245
246fn apply_merges<'tcx>(
247 body: &mut Body<'tcx>,
248 tcx: TyCtxt<'tcx>,
249 relevant: RelevantLocals,
250 merged_locals: DenseBitSet<Local>,
251) {
252 let mut merger = Merger { tcx, relevant, merged_locals };
253 merger.visit_body_preserves_cfg(body);
254}
255
256struct Merger<'tcx> {
257 tcx: TyCtxt<'tcx>,
258 relevant: RelevantLocals,
259 merged_locals: DenseBitSet<Local>,
260}
261
262impl<'tcx> MutVisitor<'tcx> for Merger<'tcx> {
263 fn tcx(&self) -> TyCtxt<'tcx> {
264 self.tcx
265 }
266
267 fn visit_local(&mut self, local: &mut Local, _: PlaceContext, _location: Location) {
268 if let Some(relevant) = self.relevant.find(*local) {
269 *local = self.relevant.original[relevant];
270 }
271 }
272
273 fn visit_statement(&mut self, statement: &mut Statement<'tcx>, location: Location) {
274 match &statement.kind {
275 // FIXME: Don't delete storage statements, but "merge" the storage ranges instead.
276 StatementKind::StorageDead(local) | StatementKind::StorageLive(local)
277 if self.merged_locals.contains(*local) =>
278 {
279 statement.make_nop();
280 return;
281 }
282 _ => (),
283 };
284 self.super_statement(statement, location);
285 match &statement.kind {
286 StatementKind::Assign(box (dest, rvalue)) => {
287 match rvalue {
288 Rvalue::CopyForDeref(place)
289 | Rvalue::Use(Operand::Copy(place) | Operand::Move(place)) => {
290 // These might've been turned into self-assignments by the replacement
291 // (this includes the original statement we wanted to eliminate).
292 if dest == place {
293 debug!("{:?} turned into self-assignment, deleting", location);
294 statement.make_nop();
295 }
296 }
297 _ => {}
298 }
299 }
300
301 _ => {}
302 }
303 }
304}
305
306//////////////////////////////////////////////////////////
307// Relevant locals
308//
309// Small utility to reduce size of the conflict matrix by only considering locals that appear in
310// the candidates
311
312newtype_index! {
313 /// Represent a subset of locals which appear in candidates.
314 struct RelevantLocal {}
315}
316
317#[derive(Debug)]
318struct RelevantLocals {
319 original: IndexVec<RelevantLocal, Local>,
320 shrink: IndexVec<Local, Option<RelevantLocal>>,
321 renames: UnionFind<RelevantLocal>,
322}
323
324impl RelevantLocals {
325 #[tracing::instrument(level = "trace", skip(candidates, num_locals), ret)]
326 fn compute(candidates: &Candidates, num_locals: usize) -> RelevantLocals {
327 let mut original = IndexVec::with_capacity(candidates.c.len());
328 let mut shrink = IndexVec::from_elem_n(None, num_locals);
329
330 // Mark a local as relevant and record it into the maps.
331 let mut declare = |local| {
332 shrink.get_or_insert_with(local, || original.push(local));
333 };
334
335 for &(src, dest) in candidates.c.iter() {
336 declare(src);
337 declare(dest)
338 }
339
340 let renames = UnionFind::new(original.len());
341 RelevantLocals { original, shrink, renames }
342 }
343
344 fn find(&mut self, src: Local) -> Option<RelevantLocal> {
345 let src = self.shrink[src]?;
346 let src = self.renames.find(src);
347 Some(src)
348 }
349
350 fn union(&mut self, lhs: RelevantLocal, rhs: RelevantLocal) -> RelevantLocal {
351 let head = self.renames.unify(lhs, rhs);
352 // We need to ensure we keep the original local of the RHS, as it may be a required local.
353 self.original[head] = self.original[rhs];
354 head
355 }
356}
357
358/////////////////////////////////////////////////////
359// Candidate accumulation
360
361#[derive(Debug, Default)]
362struct Candidates {
363 /// The set of candidates we are considering in this optimization.
364 ///
365 /// Whether a place ends up in the key or the value does not correspond to whether it appears as
366 /// the lhs or rhs of any assignment. As a matter of fact, the places in here might never appear
367 /// in an assignment at all. This happens because if we see an assignment like this:
368 ///
369 /// ```ignore (syntax-highlighting-only)
370 /// _1.0 = _2.0
371 /// ```
372 ///
373 /// We will still report that we would like to merge `_1` and `_2` in an attempt to allow us to
374 /// remove that assignment.
375 c: Vec<(Local, Local)>,
376}
377
378// We first implement some utility functions which we will expose removing candidates according to
379// different needs. Throughout the liveness filtering, the `candidates` are only ever accessed
380// through these methods, and not directly.
381impl Candidates {
382 /// Collects the candidates for merging.
383 ///
384 /// This is responsible for enforcing the first and third bullet point.
385 fn find(body: &Body<'_>, borrowed: &DenseBitSet<Local>) -> Candidates {
386 let mut visitor = FindAssignments { body, candidates: Default::default(), borrowed };
387 visitor.visit_body(body);
388
389 Candidates { c: visitor.candidates }
390 }
391}
392
393struct FindAssignments<'a, 'tcx> {
394 body: &'a Body<'tcx>,
395 candidates: Vec<(Local, Local)>,
396 borrowed: &'a DenseBitSet<Local>,
397}
398
399impl<'tcx> Visitor<'tcx> for FindAssignments<'_, 'tcx> {
400 fn visit_statement(&mut self, statement: &Statement<'tcx>, _: Location) {
401 if let StatementKind::Assign(box (
402 lhs,
403 Rvalue::CopyForDeref(rhs) | Rvalue::Use(Operand::Copy(rhs) | Operand::Move(rhs)),
404 )) = &statement.kind
405 && let Some(src) = lhs.as_local()
406 && let Some(dest) = rhs.as_local()
407 {
408 // As described at the top of the file, we do not go near things that have
409 // their address taken.
410 if self.borrowed.contains(src) || self.borrowed.contains(dest) {
411 return;
412 }
413
414 // As described at the top of this file, we do not touch locals which have
415 // different types.
416 let src_ty = self.body.local_decls()[src].ty;
417 let dest_ty = self.body.local_decls()[dest].ty;
418 if src_ty != dest_ty {
419 // FIXME(#112651): This can be removed afterwards. Also update the module description.
420 trace!("skipped `{src:?} = {dest:?}` due to subtyping: {src_ty} != {dest_ty}");
421 return;
422 }
423
424 // We may insert duplicates here, but that's fine
425 self.candidates.push((src, dest));
426 }
427 }
428}
429
430/// Some locals are part of the function's interface and can not be removed.
431///
432/// Note that these locals *can* still be merged with non-required locals by removing that other
433/// local.
434fn is_local_required(local: Local, body: &Body<'_>) -> bool {
435 match body.local_kind(local) {
436 LocalKind::Arg | LocalKind::ReturnPointer => true,
437 LocalKind::Temp => false,
438 }
439}
440
441/////////////////////////////////////////////////////////
442// MIR Dump
443
444fn dest_prop_mir_dump<'tcx>(
445 tcx: TyCtxt<'tcx>,
446 body: &Body<'tcx>,
447 points: &DenseLocationMap,
448 live: &SparseIntervalMatrix<RelevantLocal, TwoStepIndex>,
449 relevant: &RelevantLocals,
450) {
451 let locals_live_at = |location| {
452 live.rows()
453 .filter(|&r| live.contains(r, location))
454 .map(|rl| relevant.original[rl])
455 .collect::<Vec<_>>()
456 };
457
458 if let Some(dumper) = MirDumper::new(tcx, "DestinationPropagation-dataflow", body) {
459 let extra_data = &|pass_where, w: &mut dyn std::io::Write| {
460 if let PassWhere::BeforeLocation(loc) = pass_where {
461 let location = TwoStepIndex::new(points, loc, Effect::Before);
462 let live = locals_live_at(location);
463 writeln!(w, " // before: {:?} => {:?}", location, live)?;
464 }
465 if let PassWhere::AfterLocation(loc) = pass_where {
466 let location = TwoStepIndex::new(points, loc, Effect::After);
467 let live = locals_live_at(location);
468 writeln!(w, " // after: {:?} => {:?}", location, live)?;
469 }
470 Ok(())
471 };
472
473 dumper.set_extra_data(extra_data).dump_mir(body)
474 }
475}
476
477#[derive(Copy, Clone, Debug)]
478enum Effect {
479 Before,
480 After,
481}
482
483rustc_index::newtype_index! {
484 /// A reversed `PointIndex` but with the lower bit encoding early/late inside the statement.
485 /// The reversed order allows to use the more efficient `IntervalSet::append` method while we
486 /// iterate on the statements in reverse order.
487 #[orderable]
488 #[debug_format = "TwoStepIndex({})"]
489 struct TwoStepIndex {}
490}
491
492impl TwoStepIndex {
493 fn new(elements: &DenseLocationMap, location: Location, effect: Effect) -> TwoStepIndex {
494 let point = elements.point_from_location(location);
495 let effect = match effect {
496 Effect::Before => 0,
497 Effect::After => 1,
498 };
499 let max_index = 2 * elements.num_points() as u32 - 1;
500 let index = 2 * point.as_u32() + (effect as u32);
501 // Reverse the indexing to use more efficient `IntervalSet::append`.
502 TwoStepIndex::from_u32(max_index - index)
503 }
504}
505
506/// Add points depending on the result of the given dataflow analysis.
507fn save_as_intervals<'tcx>(
508 elements: &DenseLocationMap,
509 body: &Body<'tcx>,
510 relevant: &RelevantLocals,
511 results: Results<DenseBitSet<Local>>,
512) -> SparseIntervalMatrix<RelevantLocal, TwoStepIndex> {
513 let mut values = SparseIntervalMatrix::new(2 * elements.num_points());
514 let mut state = MaybeLiveLocals.bottom_value(body);
515 let reachable_blocks = traversal::reachable_as_bitset(body);
516
517 let two_step_loc = |location, effect| TwoStepIndex::new(elements, location, effect);
518 let append_at =
519 |values: &mut SparseIntervalMatrix<_, _>, state: &DenseBitSet<Local>, twostep| {
520 for (relevant, &original) in relevant.original.iter_enumerated() {
521 if state.contains(original) {
522 values.append(relevant, twostep);
523 }
524 }
525 };
526
527 // Iterate blocks in decreasing order, to visit locations in decreasing order. This
528 // allows to use the more efficient `append` method to interval sets.
529 for block in body.basic_blocks.indices().rev() {
530 if !reachable_blocks.contains(block) {
531 continue;
532 }
533
534 state.clone_from(&results[block]);
535
536 let block_data = &body.basic_blocks[block];
537 let loc = Location { block, statement_index: block_data.statements.len() };
538
539 let term = block_data.terminator();
540 let mut twostep = two_step_loc(loc, Effect::After);
541 append_at(&mut values, &state, twostep);
542 // Ensure we have a non-zero live range even for dead stores. This is done by marking all
543 // the written-to locals as live in the second half of the statement.
544 // We also ensure that operands read by terminators conflict with writes by that terminator.
545 // For instance a function call may read args after having written to the destination.
546 VisitPlacesWith(|place: Place<'tcx>, ctxt| {
547 if let Some(relevant) = relevant.shrink[place.local] {
548 match DefUse::for_place(place, ctxt) {
549 DefUse::Def | DefUse::Use | DefUse::PartialWrite => {
550 values.insert(relevant, twostep);
551 }
552 DefUse::NonUse => {}
553 }
554 }
555 })
556 .visit_terminator(term, loc);
557
558 twostep = TwoStepIndex::from_u32(twostep.as_u32() + 1);
559 debug_assert_eq!(twostep, two_step_loc(loc, Effect::Before));
560 MaybeLiveLocals.apply_early_terminator_effect(&mut state, term, loc);
561 MaybeLiveLocals.apply_primary_terminator_effect(&mut state, term, loc);
562 append_at(&mut values, &state, twostep);
563
564 for (statement_index, stmt) in block_data.statements.iter().enumerate().rev() {
565 let loc = Location { block, statement_index };
566 twostep = TwoStepIndex::from_u32(twostep.as_u32() + 1);
567 debug_assert_eq!(twostep, two_step_loc(loc, Effect::After));
568 append_at(&mut values, &state, twostep);
569 // Like terminators, ensure we have a non-zero live range even for dead stores.
570 // Some rvalues interleave reads and writes, for instance `Rvalue::Aggregate`, see
571 // https://github.com/rust-lang/rust/issues/146383. By precaution, treat statements
572 // as behaving so by default.
573 // We make an exception for simple assignments `_a.stuff = {copy|move} _b.stuff`,
574 // as marking `_b` live here would prevent unification.
575 let is_simple_assignment = match stmt.kind {
576 StatementKind::Assign(box (
577 lhs,
578 Rvalue::CopyForDeref(rhs)
579 | Rvalue::Use(Operand::Copy(rhs) | Operand::Move(rhs)),
580 )) => lhs.projection == rhs.projection,
581 _ => false,
582 };
583 VisitPlacesWith(|place: Place<'tcx>, ctxt| {
584 if let Some(relevant) = relevant.shrink[place.local] {
585 match DefUse::for_place(place, ctxt) {
586 DefUse::Def | DefUse::PartialWrite => {
587 values.insert(relevant, twostep);
588 }
589 DefUse::Use if !is_simple_assignment => {
590 values.insert(relevant, twostep);
591 }
592 DefUse::Use | DefUse::NonUse => {}
593 }
594 }
595 })
596 .visit_statement(stmt, loc);
597
598 twostep = TwoStepIndex::from_u32(twostep.as_u32() + 1);
599 debug_assert_eq!(twostep, two_step_loc(loc, Effect::Before));
600 MaybeLiveLocals.apply_early_statement_effect(&mut state, stmt, loc);
601 MaybeLiveLocals.apply_primary_statement_effect(&mut state, stmt, loc);
602 // ... but reads from operands are marked as live here so they do not conflict with
603 // the all the writes we manually marked as live in the second half of the statement.
604 append_at(&mut values, &state, twostep);
605 }
606 }
607
608 values
609}