rustc_monomorphize/
collector.rs

1//! Mono Item Collection
2//! ====================
3//!
4//! This module is responsible for discovering all items that will contribute
5//! to code generation of the crate. The important part here is that it not only
6//! needs to find syntax-level items (functions, structs, etc) but also all
7//! their monomorphized instantiations. Every non-generic, non-const function
8//! maps to one LLVM artifact. Every generic function can produce
9//! from zero to N artifacts, depending on the sets of type arguments it
10//! is instantiated with.
11//! This also applies to generic items from other crates: A generic definition
12//! in crate X might produce monomorphizations that are compiled into crate Y.
13//! We also have to collect these here.
14//!
15//! The following kinds of "mono items" are handled here:
16//!
17//! - Functions
18//! - Methods
19//! - Closures
20//! - Statics
21//! - Drop glue
22//!
23//! The following things also result in LLVM artifacts, but are not collected
24//! here, since we instantiate them locally on demand when needed in a given
25//! codegen unit:
26//!
27//! - Constants
28//! - VTables
29//! - Object Shims
30//!
31//! The main entry point is `collect_crate_mono_items`, at the bottom of this file.
32//!
33//! General Algorithm
34//! -----------------
35//! Let's define some terms first:
36//!
37//! - A "mono item" is something that results in a function or global in
38//!   the LLVM IR of a codegen unit. Mono items do not stand on their
39//!   own, they can use other mono items. For example, if function
40//!   `foo()` calls function `bar()` then the mono item for `foo()`
41//!   uses the mono item for function `bar()`. In general, the
42//!   definition for mono item A using a mono item B is that
43//!   the LLVM artifact produced for A uses the LLVM artifact produced
44//!   for B.
45//!
46//! - Mono items and the uses between them form a directed graph,
47//!   where the mono items are the nodes and uses form the edges.
48//!   Let's call this graph the "mono item graph".
49//!
50//! - The mono item graph for a program contains all mono items
51//!   that are needed in order to produce the complete LLVM IR of the program.
52//!
53//! The purpose of the algorithm implemented in this module is to build the
54//! mono item graph for the current crate. It runs in two phases:
55//!
56//! 1. Discover the roots of the graph by traversing the HIR of the crate.
57//! 2. Starting from the roots, find uses by inspecting the MIR
58//!    representation of the item corresponding to a given node, until no more
59//!    new nodes are found.
60//!
61//! ### Discovering roots
62//! The roots of the mono item graph correspond to the public non-generic
63//! syntactic items in the source code. We find them by walking the HIR of the
64//! crate, and whenever we hit upon a public function, method, or static item,
65//! we create a mono item consisting of the items DefId and, since we only
66//! consider non-generic items, an empty type-parameters set. (In eager
67//! collection mode, during incremental compilation, all non-generic functions
68//! are considered as roots, as well as when the `-Clink-dead-code` option is
69//! specified. Functions marked `#[no_mangle]` and functions called by inlinable
70//! functions also always act as roots.)
71//!
72//! ### Finding uses
73//! Given a mono item node, we can discover uses by inspecting its MIR. We walk
74//! the MIR to find other mono items used by each mono item. Since the mono
75//! item we are currently at is always monomorphic, we also know the concrete
76//! type arguments of its used mono items. The specific forms a use can take in
77//! MIR are quite diverse. Here is an overview:
78//!
79//! #### Calling Functions/Methods
80//! The most obvious way for one mono item to use another is a
81//! function or method call (represented by a CALL terminator in MIR). But
82//! calls are not the only thing that might introduce a use between two
83//! function mono items, and as we will see below, they are just a
84//! specialization of the form described next, and consequently will not get any
85//! special treatment in the algorithm.
86//!
87//! #### Taking a reference to a function or method
88//! A function does not need to actually be called in order to be used by
89//! another function. It suffices to just take a reference in order to introduce
90//! an edge. Consider the following example:
91//!
92//! ```
93//! # use core::fmt::Display;
94//! fn print_val<T: Display>(x: T) {
95//!     println!("{}", x);
96//! }
97//!
98//! fn call_fn(f: &dyn Fn(i32), x: i32) {
99//!     f(x);
100//! }
101//!
102//! fn main() {
103//!     let print_i32 = print_val::<i32>;
104//!     call_fn(&print_i32, 0);
105//! }
106//! ```
107//! The MIR of none of these functions will contain an explicit call to
108//! `print_val::<i32>`. Nonetheless, in order to mono this program, we need
109//! an instance of this function. Thus, whenever we encounter a function or
110//! method in operand position, we treat it as a use of the current
111//! mono item. Calls are just a special case of that.
112//!
113//! #### Drop glue
114//! Drop glue mono items are introduced by MIR drop-statements. The
115//! generated mono item will have additional drop-glue item uses if the
116//! type to be dropped contains nested values that also need to be dropped. It
117//! might also have a function item use for the explicit `Drop::drop`
118//! implementation of its type.
119//!
120//! #### Unsizing Casts
121//! A subtle way of introducing use edges is by casting to a trait object.
122//! Since the resulting wide-pointer contains a reference to a vtable, we need to
123//! instantiate all dyn-compatible methods of the trait, as we need to store
124//! pointers to these functions even if they never get called anywhere. This can
125//! be seen as a special case of taking a function reference.
126//!
127//!
128//! Interaction with Cross-Crate Inlining
129//! -------------------------------------
130//! The binary of a crate will not only contain machine code for the items
131//! defined in the source code of that crate. It will also contain monomorphic
132//! instantiations of any extern generic functions and of functions marked with
133//! `#[inline]`.
134//! The collection algorithm handles this more or less mono. If it is
135//! about to create a mono item for something with an external `DefId`,
136//! it will take a look if the MIR for that item is available, and if so just
137//! proceed normally. If the MIR is not available, it assumes that the item is
138//! just linked to and no node is created; which is exactly what we want, since
139//! no machine code should be generated in the current crate for such an item.
140//!
141//! Eager and Lazy Collection Strategy
142//! ----------------------------------
143//! Mono item collection can be performed with one of two strategies:
144//!
145//! - Lazy strategy means that items will only be instantiated when actually
146//!   used. The goal is to produce the least amount of machine code
147//!   possible.
148//!
149//! - Eager strategy is meant to be used in conjunction with incremental compilation
150//!   where a stable set of mono items is more important than a minimal
151//!   one. Thus, eager strategy will instantiate drop-glue for every drop-able type
152//!   in the crate, even if no drop call for that type exists (yet). It will
153//!   also instantiate default implementations of trait methods, something that
154//!   otherwise is only done on demand.
155//!
156//! Collection-time const evaluation and "mentioned" items
157//! ------------------------------------------------------
158//!
159//! One important role of collection is to evaluate all constants that are used by all the items
160//! which are being collected. Codegen can then rely on only encountering constants that evaluate
161//! successfully, and if a constant fails to evaluate, the collector has much better context to be
162//! able to show where this constant comes up.
163//!
164//! However, the exact set of "used" items (collected as described above), and therefore the exact
165//! set of used constants, can depend on optimizations. Optimizing away dead code may optimize away
166//! a function call that uses a failing constant, so an unoptimized build may fail where an
167//! optimized build succeeds. This is undesirable.
168//!
169//! To avoid this, the collector has the concept of "mentioned" items. Some time during the MIR
170//! pipeline, before any optimization-level-dependent optimizations, we compute a list of all items
171//! that syntactically appear in the code. These are considered "mentioned", and even if they are in
172//! dead code and get optimized away (which makes them no longer "used"), they are still
173//! "mentioned". For every used item, the collector ensures that all mentioned items, recursively,
174//! do not use a failing constant. This is reflected via the [`CollectionMode`], which determines
175//! whether we are visiting a used item or merely a mentioned item.
176//!
177//! The collector and "mentioned items" gathering (which lives in `rustc_mir_transform::mentioned_items`)
178//! need to stay in sync in the following sense:
179//!
180//! - For every item that the collector gather that could eventually lead to build failure (most
181//!   likely due to containing a constant that fails to evaluate), a corresponding mentioned item
182//!   must be added. This should use the exact same strategy as the ecollector to make sure they are
183//!   in sync. However, while the collector works on monomorphized types, mentioned items are
184//!   collected on generic MIR -- so any time the collector checks for a particular type (such as
185//!   `ty::FnDef`), we have to just onconditionally add this as a mentioned item.
186//! - In `visit_mentioned_item`, we then do with that mentioned item exactly what the collector
187//!   would have done during regular MIR visiting. Basically you can think of the collector having
188//!   two stages, a pre-monomorphization stage and a post-monomorphization stage (usually quite
189//!   literally separated by a call to `self.monomorphize`); the pre-monomorphizationn stage is
190//!   duplicated in mentioned items gathering and the post-monomorphization stage is duplicated in
191//!   `visit_mentioned_item`.
192//! - Finally, as a performance optimization, the collector should fill `used_mentioned_item` during
193//!   its MIR traversal with exactly what mentioned item gathering would have added in the same
194//!   situation. This detects mentioned items that have *not* been optimized away and hence don't
195//!   need a dedicated traversal.
196//!
197//! Open Issues
198//! -----------
199//! Some things are not yet fully implemented in the current version of this
200//! module.
201//!
202//! ### Const Fns
203//! Ideally, no mono item should be generated for const fns unless there
204//! is a call to them that cannot be evaluated at compile time. At the moment
205//! this is not implemented however: a mono item will be produced
206//! regardless of whether it is actually needed or not.
207
208use std::cell::OnceCell;
209use std::path::PathBuf;
210
211use rustc_data_structures::fx::FxIndexMap;
212use rustc_data_structures::sync::{MTLock, par_for_each_in};
213use rustc_data_structures::unord::{UnordMap, UnordSet};
214use rustc_hir as hir;
215use rustc_hir::attrs::InlineAttr;
216use rustc_hir::def::DefKind;
217use rustc_hir::def_id::{DefId, DefIdMap, LocalDefId};
218use rustc_hir::lang_items::LangItem;
219use rustc_middle::middle::codegen_fn_attrs::CodegenFnAttrFlags;
220use rustc_middle::mir::interpret::{AllocId, ErrorHandled, GlobalAlloc, Scalar};
221use rustc_middle::mir::mono::{CollectionMode, InstantiationMode, MonoItem};
222use rustc_middle::mir::visit::Visitor as MirVisitor;
223use rustc_middle::mir::{self, Location, MentionedItem, traversal};
224use rustc_middle::query::TyCtxtAt;
225use rustc_middle::ty::adjustment::{CustomCoerceUnsized, PointerCoercion};
226use rustc_middle::ty::layout::ValidityRequirement;
227use rustc_middle::ty::print::{shrunk_instance_name, with_no_trimmed_paths};
228use rustc_middle::ty::{
229    self, GenericArgs, GenericParamDefKind, Instance, InstanceKind, Ty, TyCtxt, TypeFoldable,
230    TypeVisitableExt, VtblEntry,
231};
232use rustc_middle::util::Providers;
233use rustc_middle::{bug, span_bug};
234use rustc_session::Limit;
235use rustc_session::config::{DebugInfo, EntryFnType};
236use rustc_span::source_map::{Spanned, dummy_spanned, respan};
237use rustc_span::{DUMMY_SP, Span};
238use tracing::{debug, instrument, trace};
239
240use crate::errors::{self, EncounteredErrorWhileInstantiating, NoOptimizedMir, RecursionLimit};
241
242#[derive(PartialEq)]
243pub(crate) enum MonoItemCollectionStrategy {
244    Eager,
245    Lazy,
246}
247
248/// The state that is shared across the concurrent threads that are doing collection.
249struct SharedState<'tcx> {
250    /// Items that have been or are currently being recursively collected.
251    visited: MTLock<UnordSet<MonoItem<'tcx>>>,
252    /// Items that have been or are currently being recursively treated as "mentioned", i.e., their
253    /// consts are evaluated but nothing is added to the collection.
254    mentioned: MTLock<UnordSet<MonoItem<'tcx>>>,
255    /// Which items are being used where, for better errors.
256    usage_map: MTLock<UsageMap<'tcx>>,
257}
258
259pub(crate) struct UsageMap<'tcx> {
260    // Maps every mono item to the mono items used by it.
261    pub used_map: UnordMap<MonoItem<'tcx>, Vec<MonoItem<'tcx>>>,
262
263    // Maps every mono item to the mono items that use it.
264    user_map: UnordMap<MonoItem<'tcx>, Vec<MonoItem<'tcx>>>,
265}
266
267impl<'tcx> UsageMap<'tcx> {
268    fn new() -> UsageMap<'tcx> {
269        UsageMap { used_map: Default::default(), user_map: Default::default() }
270    }
271
272    fn record_used<'a>(&mut self, user_item: MonoItem<'tcx>, used_items: &'a MonoItems<'tcx>)
273    where
274        'tcx: 'a,
275    {
276        for used_item in used_items.items() {
277            self.user_map.entry(used_item).or_default().push(user_item);
278        }
279
280        assert!(self.used_map.insert(user_item, used_items.items().collect()).is_none());
281    }
282
283    pub(crate) fn get_user_items(&self, item: MonoItem<'tcx>) -> &[MonoItem<'tcx>] {
284        self.user_map.get(&item).map(|items| items.as_slice()).unwrap_or(&[])
285    }
286
287    /// Internally iterate over all inlined items used by `item`.
288    pub(crate) fn for_each_inlined_used_item<F>(
289        &self,
290        tcx: TyCtxt<'tcx>,
291        item: MonoItem<'tcx>,
292        mut f: F,
293    ) where
294        F: FnMut(MonoItem<'tcx>),
295    {
296        let used_items = self.used_map.get(&item).unwrap();
297        for used_item in used_items.iter() {
298            let is_inlined = used_item.instantiation_mode(tcx) == InstantiationMode::LocalCopy;
299            if is_inlined {
300                f(*used_item);
301            }
302        }
303    }
304}
305
306struct MonoItems<'tcx> {
307    // We want a set of MonoItem + Span where trying to re-insert a MonoItem with a different Span
308    // is ignored. Map does that, but it looks odd.
309    items: FxIndexMap<MonoItem<'tcx>, Span>,
310}
311
312impl<'tcx> MonoItems<'tcx> {
313    fn new() -> Self {
314        Self { items: FxIndexMap::default() }
315    }
316
317    fn is_empty(&self) -> bool {
318        self.items.is_empty()
319    }
320
321    fn push(&mut self, item: Spanned<MonoItem<'tcx>>) {
322        // Insert only if the entry does not exist. A normal insert would stomp the first span that
323        // got inserted.
324        self.items.entry(item.node).or_insert(item.span);
325    }
326
327    fn items(&self) -> impl Iterator<Item = MonoItem<'tcx>> {
328        self.items.keys().cloned()
329    }
330}
331
332impl<'tcx> IntoIterator for MonoItems<'tcx> {
333    type Item = Spanned<MonoItem<'tcx>>;
334    type IntoIter = impl Iterator<Item = Spanned<MonoItem<'tcx>>>;
335
336    fn into_iter(self) -> Self::IntoIter {
337        self.items.into_iter().map(|(item, span)| respan(span, item))
338    }
339}
340
341impl<'tcx> Extend<Spanned<MonoItem<'tcx>>> for MonoItems<'tcx> {
342    fn extend<I>(&mut self, iter: I)
343    where
344        I: IntoIterator<Item = Spanned<MonoItem<'tcx>>>,
345    {
346        for item in iter {
347            self.push(item)
348        }
349    }
350}
351
352fn collect_items_root<'tcx>(
353    tcx: TyCtxt<'tcx>,
354    starting_item: Spanned<MonoItem<'tcx>>,
355    state: &SharedState<'tcx>,
356    recursion_limit: Limit,
357) {
358    if !state.visited.lock_mut().insert(starting_item.node) {
359        // We've been here already, no need to search again.
360        return;
361    }
362    let mut recursion_depths = DefIdMap::default();
363    collect_items_rec(
364        tcx,
365        starting_item,
366        state,
367        &mut recursion_depths,
368        recursion_limit,
369        CollectionMode::UsedItems,
370    );
371}
372
373/// Collect all monomorphized items reachable from `starting_point`, and emit a note diagnostic if a
374/// post-monomorphization error is encountered during a collection step.
375///
376/// `mode` determined whether we are scanning for [used items][CollectionMode::UsedItems]
377/// or [mentioned items][CollectionMode::MentionedItems].
378#[instrument(skip(tcx, state, recursion_depths, recursion_limit), level = "debug")]
379fn collect_items_rec<'tcx>(
380    tcx: TyCtxt<'tcx>,
381    starting_item: Spanned<MonoItem<'tcx>>,
382    state: &SharedState<'tcx>,
383    recursion_depths: &mut DefIdMap<usize>,
384    recursion_limit: Limit,
385    mode: CollectionMode,
386) {
387    let mut used_items = MonoItems::new();
388    let mut mentioned_items = MonoItems::new();
389    let recursion_depth_reset;
390
391    // Post-monomorphization errors MVP
392    //
393    // We can encounter errors while monomorphizing an item, but we don't have a good way of
394    // showing a complete stack of spans ultimately leading to collecting the erroneous one yet.
395    // (It's also currently unclear exactly which diagnostics and information would be interesting
396    // to report in such cases)
397    //
398    // This leads to suboptimal error reporting: a post-monomorphization error (PME) will be
399    // shown with just a spanned piece of code causing the error, without information on where
400    // it was called from. This is especially obscure if the erroneous mono item is in a
401    // dependency. See for example issue #85155, where, before minimization, a PME happened two
402    // crates downstream from libcore's stdarch, without a way to know which dependency was the
403    // cause.
404    //
405    // If such an error occurs in the current crate, its span will be enough to locate the
406    // source. If the cause is in another crate, the goal here is to quickly locate which mono
407    // item in the current crate is ultimately responsible for causing the error.
408    //
409    // To give at least _some_ context to the user: while collecting mono items, we check the
410    // error count. If it has changed, a PME occurred, and we trigger some diagnostics about the
411    // current step of mono items collection.
412    //
413    // FIXME: don't rely on global state, instead bubble up errors. Note: this is very hard to do.
414    let error_count = tcx.dcx().err_count();
415
416    // In `mentioned_items` we collect items that were mentioned in this MIR but possibly do not
417    // need to be monomorphized. This is done to ensure that optimizing away function calls does not
418    // hide const-eval errors that those calls would otherwise have triggered.
419    match starting_item.node {
420        MonoItem::Static(def_id) => {
421            recursion_depth_reset = None;
422
423            // Statics always get evaluated (which is possible because they can't be generic), so for
424            // `MentionedItems` collection there's nothing to do here.
425            if mode == CollectionMode::UsedItems {
426                let instance = Instance::mono(tcx, def_id);
427
428                // Sanity check whether this ended up being collected accidentally
429                debug_assert!(tcx.should_codegen_locally(instance));
430
431                let DefKind::Static { nested, .. } = tcx.def_kind(def_id) else { bug!() };
432                // Nested statics have no type.
433                if !nested {
434                    let ty = instance.ty(tcx, ty::TypingEnv::fully_monomorphized());
435                    visit_drop_use(tcx, ty, true, starting_item.span, &mut used_items);
436                }
437
438                if let Ok(alloc) = tcx.eval_static_initializer(def_id) {
439                    for &prov in alloc.inner().provenance().ptrs().values() {
440                        collect_alloc(tcx, prov.alloc_id(), &mut used_items);
441                    }
442                }
443
444                if tcx.needs_thread_local_shim(def_id) {
445                    used_items.push(respan(
446                        starting_item.span,
447                        MonoItem::Fn(Instance {
448                            def: InstanceKind::ThreadLocalShim(def_id),
449                            args: GenericArgs::empty(),
450                        }),
451                    ));
452                }
453            }
454
455            // mentioned_items stays empty since there's no codegen for statics. statics don't get
456            // optimized, and if they did then the const-eval interpreter would have to worry about
457            // mentioned_items.
458        }
459        MonoItem::Fn(instance) => {
460            // Sanity check whether this ended up being collected accidentally
461            debug_assert!(tcx.should_codegen_locally(instance));
462
463            // Keep track of the monomorphization recursion depth
464            recursion_depth_reset = Some(check_recursion_limit(
465                tcx,
466                instance,
467                starting_item.span,
468                recursion_depths,
469                recursion_limit,
470            ));
471
472            rustc_data_structures::stack::ensure_sufficient_stack(|| {
473                let (used, mentioned) = tcx.items_of_instance((instance, mode));
474                used_items.extend(used.into_iter().copied());
475                mentioned_items.extend(mentioned.into_iter().copied());
476            });
477        }
478        MonoItem::GlobalAsm(item_id) => {
479            assert!(
480                mode == CollectionMode::UsedItems,
481                "should never encounter global_asm when collecting mentioned items"
482            );
483            recursion_depth_reset = None;
484
485            let item = tcx.hir_item(item_id);
486            if let hir::ItemKind::GlobalAsm { asm, .. } = item.kind {
487                for (op, op_sp) in asm.operands {
488                    match *op {
489                        hir::InlineAsmOperand::Const { .. } => {
490                            // Only constants which resolve to a plain integer
491                            // are supported. Therefore the value should not
492                            // depend on any other items.
493                        }
494                        hir::InlineAsmOperand::SymFn { expr } => {
495                            let fn_ty = tcx.typeck(item_id.owner_id).expr_ty(expr);
496                            visit_fn_use(tcx, fn_ty, false, *op_sp, &mut used_items);
497                        }
498                        hir::InlineAsmOperand::SymStatic { path: _, def_id } => {
499                            let instance = Instance::mono(tcx, def_id);
500                            if tcx.should_codegen_locally(instance) {
501                                trace!("collecting static {:?}", def_id);
502                                used_items.push(dummy_spanned(MonoItem::Static(def_id)));
503                            }
504                        }
505                        hir::InlineAsmOperand::In { .. }
506                        | hir::InlineAsmOperand::Out { .. }
507                        | hir::InlineAsmOperand::InOut { .. }
508                        | hir::InlineAsmOperand::SplitInOut { .. }
509                        | hir::InlineAsmOperand::Label { .. } => {
510                            span_bug!(*op_sp, "invalid operand type for global_asm!")
511                        }
512                    }
513                }
514            } else {
515                span_bug!(item.span, "Mismatch between hir::Item type and MonoItem type")
516            }
517
518            // mention_items stays empty as nothing gets optimized here.
519        }
520    };
521
522    // Check for PMEs and emit a diagnostic if one happened. To try to show relevant edges of the
523    // mono item graph.
524    if tcx.dcx().err_count() > error_count
525        && starting_item.node.is_generic_fn()
526        && starting_item.node.is_user_defined()
527    {
528        let formatted_item = with_no_trimmed_paths!(starting_item.node.to_string());
529        tcx.dcx().emit_note(EncounteredErrorWhileInstantiating {
530            span: starting_item.span,
531            formatted_item,
532        });
533    }
534    // Only updating `usage_map` for used items as otherwise we may be inserting the same item
535    // multiple times (if it is first 'mentioned' and then later actually used), and the usage map
536    // logic does not like that.
537    // This is part of the output of collection and hence only relevant for "used" items.
538    // ("Mentioned" items are only considered internally during collection.)
539    if mode == CollectionMode::UsedItems {
540        state.usage_map.lock_mut().record_used(starting_item.node, &used_items);
541    }
542
543    {
544        let mut visited = OnceCell::default();
545        if mode == CollectionMode::UsedItems {
546            used_items
547                .items
548                .retain(|k, _| visited.get_mut_or_init(|| state.visited.lock_mut()).insert(*k));
549        }
550
551        let mut mentioned = OnceCell::default();
552        mentioned_items.items.retain(|k, _| {
553            !visited.get_or_init(|| state.visited.lock()).contains(k)
554                && mentioned.get_mut_or_init(|| state.mentioned.lock_mut()).insert(*k)
555        });
556    }
557    if mode == CollectionMode::MentionedItems {
558        assert!(used_items.is_empty(), "'mentioned' collection should never encounter used items");
559    } else {
560        for used_item in used_items {
561            collect_items_rec(
562                tcx,
563                used_item,
564                state,
565                recursion_depths,
566                recursion_limit,
567                CollectionMode::UsedItems,
568            );
569        }
570    }
571
572    // Walk over mentioned items *after* used items, so that if an item is both mentioned and used then
573    // the loop above has fully collected it, so this loop will skip it.
574    for mentioned_item in mentioned_items {
575        collect_items_rec(
576            tcx,
577            mentioned_item,
578            state,
579            recursion_depths,
580            recursion_limit,
581            CollectionMode::MentionedItems,
582        );
583    }
584
585    if let Some((def_id, depth)) = recursion_depth_reset {
586        recursion_depths.insert(def_id, depth);
587    }
588}
589
590fn check_recursion_limit<'tcx>(
591    tcx: TyCtxt<'tcx>,
592    instance: Instance<'tcx>,
593    span: Span,
594    recursion_depths: &mut DefIdMap<usize>,
595    recursion_limit: Limit,
596) -> (DefId, usize) {
597    let def_id = instance.def_id();
598    let recursion_depth = recursion_depths.get(&def_id).cloned().unwrap_or(0);
599    debug!(" => recursion depth={}", recursion_depth);
600
601    let adjusted_recursion_depth = if tcx.is_lang_item(def_id, LangItem::DropInPlace) {
602        // HACK: drop_in_place creates tight monomorphization loops. Give
603        // it more margin.
604        recursion_depth / 4
605    } else {
606        recursion_depth
607    };
608
609    // Code that needs to instantiate the same function recursively
610    // more than the recursion limit is assumed to be causing an
611    // infinite expansion.
612    if !recursion_limit.value_within_limit(adjusted_recursion_depth) {
613        let def_span = tcx.def_span(def_id);
614        let def_path_str = tcx.def_path_str(def_id);
615        let (shrunk, written_to_path) = shrunk_instance_name(tcx, instance);
616        let mut path = PathBuf::new();
617        let was_written = if let Some(written_to_path) = written_to_path {
618            path = written_to_path;
619            true
620        } else {
621            false
622        };
623        tcx.dcx().emit_fatal(RecursionLimit {
624            span,
625            shrunk,
626            def_span,
627            def_path_str,
628            was_written,
629            path,
630        });
631    }
632
633    recursion_depths.insert(def_id, recursion_depth + 1);
634
635    (def_id, recursion_depth)
636}
637
638struct MirUsedCollector<'a, 'tcx> {
639    tcx: TyCtxt<'tcx>,
640    body: &'a mir::Body<'tcx>,
641    used_items: &'a mut MonoItems<'tcx>,
642    /// See the comment in `collect_items_of_instance` for the purpose of this set.
643    /// Note that this contains *not-monomorphized* items!
644    used_mentioned_items: &'a mut UnordSet<MentionedItem<'tcx>>,
645    instance: Instance<'tcx>,
646}
647
648impl<'a, 'tcx> MirUsedCollector<'a, 'tcx> {
649    fn monomorphize<T>(&self, value: T) -> T
650    where
651        T: TypeFoldable<TyCtxt<'tcx>>,
652    {
653        trace!("monomorphize: self.instance={:?}", self.instance);
654        self.instance.instantiate_mir_and_normalize_erasing_regions(
655            self.tcx,
656            ty::TypingEnv::fully_monomorphized(),
657            ty::EarlyBinder::bind(value),
658        )
659    }
660
661    /// Evaluates a *not yet monomorphized* constant.
662    fn eval_constant(&mut self, constant: &mir::ConstOperand<'tcx>) -> Option<mir::ConstValue> {
663        let const_ = self.monomorphize(constant.const_);
664        // Evaluate the constant. This makes const eval failure a collection-time error (rather than
665        // a codegen-time error). rustc stops after collection if there was an error, so this
666        // ensures codegen never has to worry about failing consts.
667        // (codegen relies on this and ICEs will happen if this is violated.)
668        match const_.eval(self.tcx, ty::TypingEnv::fully_monomorphized(), constant.span) {
669            Ok(v) => Some(v),
670            Err(ErrorHandled::TooGeneric(..)) => span_bug!(
671                constant.span,
672                "collection encountered polymorphic constant: {:?}",
673                const_
674            ),
675            Err(err @ ErrorHandled::Reported(..)) => {
676                err.emit_note(self.tcx);
677                return None;
678            }
679        }
680    }
681}
682
683impl<'a, 'tcx> MirVisitor<'tcx> for MirUsedCollector<'a, 'tcx> {
684    fn visit_rvalue(&mut self, rvalue: &mir::Rvalue<'tcx>, location: Location) {
685        debug!("visiting rvalue {:?}", *rvalue);
686
687        let span = self.body.source_info(location).span;
688
689        match *rvalue {
690            // When doing an cast from a regular pointer to a wide pointer, we
691            // have to instantiate all methods of the trait being cast to, so we
692            // can build the appropriate vtable.
693            mir::Rvalue::Cast(
694                mir::CastKind::PointerCoercion(PointerCoercion::Unsize, _),
695                ref operand,
696                target_ty,
697            ) => {
698                let source_ty = operand.ty(self.body, self.tcx);
699                // *Before* monomorphizing, record that we already handled this mention.
700                self.used_mentioned_items
701                    .insert(MentionedItem::UnsizeCast { source_ty, target_ty });
702                let target_ty = self.monomorphize(target_ty);
703                let source_ty = self.monomorphize(source_ty);
704                let (source_ty, target_ty) =
705                    find_tails_for_unsizing(self.tcx.at(span), source_ty, target_ty);
706                // This could also be a different Unsize instruction, like
707                // from a fixed sized array to a slice. But we are only
708                // interested in things that produce a vtable.
709                if target_ty.is_trait() && !source_ty.is_trait() {
710                    create_mono_items_for_vtable_methods(
711                        self.tcx,
712                        target_ty,
713                        source_ty,
714                        span,
715                        self.used_items,
716                    );
717                }
718            }
719            mir::Rvalue::Cast(
720                mir::CastKind::PointerCoercion(PointerCoercion::ReifyFnPointer, _),
721                ref operand,
722                _,
723            ) => {
724                let fn_ty = operand.ty(self.body, self.tcx);
725                // *Before* monomorphizing, record that we already handled this mention.
726                self.used_mentioned_items.insert(MentionedItem::Fn(fn_ty));
727                let fn_ty = self.monomorphize(fn_ty);
728                visit_fn_use(self.tcx, fn_ty, false, span, self.used_items);
729            }
730            mir::Rvalue::Cast(
731                mir::CastKind::PointerCoercion(PointerCoercion::ClosureFnPointer(_), _),
732                ref operand,
733                _,
734            ) => {
735                let source_ty = operand.ty(self.body, self.tcx);
736                // *Before* monomorphizing, record that we already handled this mention.
737                self.used_mentioned_items.insert(MentionedItem::Closure(source_ty));
738                let source_ty = self.monomorphize(source_ty);
739                if let ty::Closure(def_id, args) = *source_ty.kind() {
740                    let instance =
741                        Instance::resolve_closure(self.tcx, def_id, args, ty::ClosureKind::FnOnce);
742                    if self.tcx.should_codegen_locally(instance) {
743                        self.used_items.push(create_fn_mono_item(self.tcx, instance, span));
744                    }
745                } else {
746                    bug!()
747                }
748            }
749            mir::Rvalue::ThreadLocalRef(def_id) => {
750                assert!(self.tcx.is_thread_local_static(def_id));
751                let instance = Instance::mono(self.tcx, def_id);
752                if self.tcx.should_codegen_locally(instance) {
753                    trace!("collecting thread-local static {:?}", def_id);
754                    self.used_items.push(respan(span, MonoItem::Static(def_id)));
755                }
756            }
757            _ => { /* not interesting */ }
758        }
759
760        self.super_rvalue(rvalue, location);
761    }
762
763    /// This does not walk the MIR of the constant as that is not needed for codegen, all we need is
764    /// to ensure that the constant evaluates successfully and walk the result.
765    #[instrument(skip(self), level = "debug")]
766    fn visit_const_operand(&mut self, constant: &mir::ConstOperand<'tcx>, _location: Location) {
767        // No `super_constant` as we don't care about `visit_ty`/`visit_ty_const`.
768        let Some(val) = self.eval_constant(constant) else { return };
769        collect_const_value(self.tcx, val, self.used_items);
770    }
771
772    fn visit_terminator(&mut self, terminator: &mir::Terminator<'tcx>, location: Location) {
773        debug!("visiting terminator {:?} @ {:?}", terminator, location);
774        let source = self.body.source_info(location).span;
775
776        let tcx = self.tcx;
777        let push_mono_lang_item = |this: &mut Self, lang_item: LangItem| {
778            let instance = Instance::mono(tcx, tcx.require_lang_item(lang_item, source));
779            if tcx.should_codegen_locally(instance) {
780                this.used_items.push(create_fn_mono_item(tcx, instance, source));
781            }
782        };
783
784        match terminator.kind {
785            mir::TerminatorKind::Call { ref func, .. }
786            | mir::TerminatorKind::TailCall { ref func, .. } => {
787                let callee_ty = func.ty(self.body, tcx);
788                // *Before* monomorphizing, record that we already handled this mention.
789                self.used_mentioned_items.insert(MentionedItem::Fn(callee_ty));
790                let callee_ty = self.monomorphize(callee_ty);
791                visit_fn_use(self.tcx, callee_ty, true, source, &mut self.used_items)
792            }
793            mir::TerminatorKind::Drop { ref place, .. } => {
794                let ty = place.ty(self.body, self.tcx).ty;
795                // *Before* monomorphizing, record that we already handled this mention.
796                self.used_mentioned_items.insert(MentionedItem::Drop(ty));
797                let ty = self.monomorphize(ty);
798                visit_drop_use(self.tcx, ty, true, source, self.used_items);
799            }
800            mir::TerminatorKind::InlineAsm { ref operands, .. } => {
801                for op in operands {
802                    match *op {
803                        mir::InlineAsmOperand::SymFn { ref value } => {
804                            let fn_ty = value.const_.ty();
805                            // *Before* monomorphizing, record that we already handled this mention.
806                            self.used_mentioned_items.insert(MentionedItem::Fn(fn_ty));
807                            let fn_ty = self.monomorphize(fn_ty);
808                            visit_fn_use(self.tcx, fn_ty, false, source, self.used_items);
809                        }
810                        mir::InlineAsmOperand::SymStatic { def_id } => {
811                            let instance = Instance::mono(self.tcx, def_id);
812                            if self.tcx.should_codegen_locally(instance) {
813                                trace!("collecting asm sym static {:?}", def_id);
814                                self.used_items.push(respan(source, MonoItem::Static(def_id)));
815                            }
816                        }
817                        _ => {}
818                    }
819                }
820            }
821            mir::TerminatorKind::Assert { ref msg, .. } => match &**msg {
822                mir::AssertKind::BoundsCheck { .. } => {
823                    push_mono_lang_item(self, LangItem::PanicBoundsCheck);
824                }
825                mir::AssertKind::MisalignedPointerDereference { .. } => {
826                    push_mono_lang_item(self, LangItem::PanicMisalignedPointerDereference);
827                }
828                mir::AssertKind::NullPointerDereference => {
829                    push_mono_lang_item(self, LangItem::PanicNullPointerDereference);
830                }
831                mir::AssertKind::InvalidEnumConstruction(_) => {
832                    push_mono_lang_item(self, LangItem::PanicInvalidEnumConstruction);
833                }
834                _ => {
835                    push_mono_lang_item(self, msg.panic_function());
836                }
837            },
838            mir::TerminatorKind::UnwindTerminate(reason) => {
839                push_mono_lang_item(self, reason.lang_item());
840            }
841            mir::TerminatorKind::Goto { .. }
842            | mir::TerminatorKind::SwitchInt { .. }
843            | mir::TerminatorKind::UnwindResume
844            | mir::TerminatorKind::Return
845            | mir::TerminatorKind::Unreachable => {}
846            mir::TerminatorKind::CoroutineDrop
847            | mir::TerminatorKind::Yield { .. }
848            | mir::TerminatorKind::FalseEdge { .. }
849            | mir::TerminatorKind::FalseUnwind { .. } => bug!(),
850        }
851
852        if let Some(mir::UnwindAction::Terminate(reason)) = terminator.unwind() {
853            push_mono_lang_item(self, reason.lang_item());
854        }
855
856        self.super_terminator(terminator, location);
857    }
858}
859
860fn visit_drop_use<'tcx>(
861    tcx: TyCtxt<'tcx>,
862    ty: Ty<'tcx>,
863    is_direct_call: bool,
864    source: Span,
865    output: &mut MonoItems<'tcx>,
866) {
867    let instance = Instance::resolve_drop_in_place(tcx, ty);
868    visit_instance_use(tcx, instance, is_direct_call, source, output);
869}
870
871/// For every call of this function in the visitor, make sure there is a matching call in the
872/// `mentioned_items` pass!
873fn visit_fn_use<'tcx>(
874    tcx: TyCtxt<'tcx>,
875    ty: Ty<'tcx>,
876    is_direct_call: bool,
877    source: Span,
878    output: &mut MonoItems<'tcx>,
879) {
880    if let ty::FnDef(def_id, args) = *ty.kind() {
881        let instance = if is_direct_call {
882            ty::Instance::expect_resolve(
883                tcx,
884                ty::TypingEnv::fully_monomorphized(),
885                def_id,
886                args,
887                source,
888            )
889        } else {
890            match ty::Instance::resolve_for_fn_ptr(
891                tcx,
892                ty::TypingEnv::fully_monomorphized(),
893                def_id,
894                args,
895            ) {
896                Some(instance) => instance,
897                _ => bug!("failed to resolve instance for {ty}"),
898            }
899        };
900        visit_instance_use(tcx, instance, is_direct_call, source, output);
901    }
902}
903
904fn visit_instance_use<'tcx>(
905    tcx: TyCtxt<'tcx>,
906    instance: ty::Instance<'tcx>,
907    is_direct_call: bool,
908    source: Span,
909    output: &mut MonoItems<'tcx>,
910) {
911    debug!("visit_item_use({:?}, is_direct_call={:?})", instance, is_direct_call);
912    if !tcx.should_codegen_locally(instance) {
913        return;
914    }
915    if let Some(intrinsic) = tcx.intrinsic(instance.def_id()) {
916        if let Some(_requirement) = ValidityRequirement::from_intrinsic(intrinsic.name) {
917            // The intrinsics assert_inhabited, assert_zero_valid, and assert_mem_uninitialized_valid will
918            // be lowered in codegen to nothing or a call to panic_nounwind. So if we encounter any
919            // of those intrinsics, we need to include a mono item for panic_nounwind, else we may try to
920            // codegen a call to that function without generating code for the function itself.
921            let def_id = tcx.require_lang_item(LangItem::PanicNounwind, source);
922            let panic_instance = Instance::mono(tcx, def_id);
923            if tcx.should_codegen_locally(panic_instance) {
924                output.push(create_fn_mono_item(tcx, panic_instance, source));
925            }
926        } else if !intrinsic.must_be_overridden {
927            // Codegen the fallback body of intrinsics with fallback bodies.
928            // We explicitly skip this otherwise to ensure we get a linker error
929            // if anyone tries to call this intrinsic and the codegen backend did not
930            // override the implementation.
931            let instance = ty::Instance::new_raw(instance.def_id(), instance.args);
932            if tcx.should_codegen_locally(instance) {
933                output.push(create_fn_mono_item(tcx, instance, source));
934            }
935        }
936    }
937
938    match instance.def {
939        ty::InstanceKind::Virtual(..) | ty::InstanceKind::Intrinsic(_) => {
940            if !is_direct_call {
941                bug!("{:?} being reified", instance);
942            }
943        }
944        ty::InstanceKind::ThreadLocalShim(..) => {
945            bug!("{:?} being reified", instance);
946        }
947        ty::InstanceKind::DropGlue(_, None) => {
948            // Don't need to emit noop drop glue if we are calling directly.
949            //
950            // Note that we also optimize away the call to visit_instance_use in vtable construction
951            // (see create_mono_items_for_vtable_methods).
952            if !is_direct_call {
953                output.push(create_fn_mono_item(tcx, instance, source));
954            }
955        }
956        ty::InstanceKind::DropGlue(_, Some(_))
957        | ty::InstanceKind::FutureDropPollShim(..)
958        | ty::InstanceKind::AsyncDropGlue(_, _)
959        | ty::InstanceKind::AsyncDropGlueCtorShim(_, _)
960        | ty::InstanceKind::VTableShim(..)
961        | ty::InstanceKind::ReifyShim(..)
962        | ty::InstanceKind::ClosureOnceShim { .. }
963        | ty::InstanceKind::ConstructCoroutineInClosureShim { .. }
964        | ty::InstanceKind::Item(..)
965        | ty::InstanceKind::FnPtrShim(..)
966        | ty::InstanceKind::CloneShim(..)
967        | ty::InstanceKind::FnPtrAddrShim(..) => {
968            output.push(create_fn_mono_item(tcx, instance, source));
969        }
970    }
971}
972
973/// Returns `true` if we should codegen an instance in the local crate, or returns `false` if we
974/// can just link to the upstream crate and therefore don't need a mono item.
975fn should_codegen_locally<'tcx>(tcx: TyCtxt<'tcx>, instance: Instance<'tcx>) -> bool {
976    let Some(def_id) = instance.def.def_id_if_not_guaranteed_local_codegen() else {
977        return true;
978    };
979
980    if tcx.is_foreign_item(def_id) {
981        // Foreign items are always linked against, there's no way of instantiating them.
982        return false;
983    }
984
985    if tcx.def_kind(def_id).has_codegen_attrs()
986        && matches!(tcx.codegen_fn_attrs(def_id).inline, InlineAttr::Force { .. })
987    {
988        // `#[rustc_force_inline]` items should never be codegened. This should be caught by
989        // the MIR validator.
990        tcx.dcx().delayed_bug("attempt to codegen `#[rustc_force_inline]` item");
991    }
992
993    if def_id.is_local() {
994        // Local items cannot be referred to locally without monomorphizing them locally.
995        return true;
996    }
997
998    if tcx.is_reachable_non_generic(def_id) || instance.upstream_monomorphization(tcx).is_some() {
999        // We can link to the item in question, no instance needed in this crate.
1000        return false;
1001    }
1002
1003    if let DefKind::Static { .. } = tcx.def_kind(def_id) {
1004        // We cannot monomorphize statics from upstream crates.
1005        return false;
1006    }
1007
1008    if !tcx.is_mir_available(def_id) {
1009        tcx.dcx().emit_fatal(NoOptimizedMir {
1010            span: tcx.def_span(def_id),
1011            crate_name: tcx.crate_name(def_id.krate),
1012            instance: instance.to_string(),
1013        });
1014    }
1015
1016    true
1017}
1018
1019/// For a given pair of source and target type that occur in an unsizing coercion,
1020/// this function finds the pair of types that determines the vtable linking
1021/// them.
1022///
1023/// For example, the source type might be `&SomeStruct` and the target type
1024/// might be `&dyn SomeTrait` in a cast like:
1025///
1026/// ```rust,ignore (not real code)
1027/// let src: &SomeStruct = ...;
1028/// let target = src as &dyn SomeTrait;
1029/// ```
1030///
1031/// Then the output of this function would be (SomeStruct, SomeTrait) since for
1032/// constructing the `target` wide-pointer we need the vtable for that pair.
1033///
1034/// Things can get more complicated though because there's also the case where
1035/// the unsized type occurs as a field:
1036///
1037/// ```rust
1038/// struct ComplexStruct<T: ?Sized> {
1039///    a: u32,
1040///    b: f64,
1041///    c: T
1042/// }
1043/// ```
1044///
1045/// In this case, if `T` is sized, `&ComplexStruct<T>` is a thin pointer. If `T`
1046/// is unsized, `&SomeStruct` is a wide pointer, and the vtable it points to is
1047/// for the pair of `T` (which is a trait) and the concrete type that `T` was
1048/// originally coerced from:
1049///
1050/// ```rust,ignore (not real code)
1051/// let src: &ComplexStruct<SomeStruct> = ...;
1052/// let target = src as &ComplexStruct<dyn SomeTrait>;
1053/// ```
1054///
1055/// Again, we want this `find_vtable_types_for_unsizing()` to provide the pair
1056/// `(SomeStruct, SomeTrait)`.
1057///
1058/// Finally, there is also the case of custom unsizing coercions, e.g., for
1059/// smart pointers such as `Rc` and `Arc`.
1060fn find_tails_for_unsizing<'tcx>(
1061    tcx: TyCtxtAt<'tcx>,
1062    source_ty: Ty<'tcx>,
1063    target_ty: Ty<'tcx>,
1064) -> (Ty<'tcx>, Ty<'tcx>) {
1065    let typing_env = ty::TypingEnv::fully_monomorphized();
1066    debug_assert!(!source_ty.has_param(), "{source_ty} should be fully monomorphic");
1067    debug_assert!(!target_ty.has_param(), "{target_ty} should be fully monomorphic");
1068
1069    match (source_ty.kind(), target_ty.kind()) {
1070        (
1071            &ty::Ref(_, source_pointee, _),
1072            &ty::Ref(_, target_pointee, _) | &ty::RawPtr(target_pointee, _),
1073        )
1074        | (&ty::RawPtr(source_pointee, _), &ty::RawPtr(target_pointee, _)) => {
1075            tcx.struct_lockstep_tails_for_codegen(source_pointee, target_pointee, typing_env)
1076        }
1077
1078        // `Box<T>` could go through the ADT code below, b/c it'll unpeel to `Unique<T>`,
1079        // and eventually bottom out in a raw ref, but we can micro-optimize it here.
1080        (_, _)
1081            if let Some(source_boxed) = source_ty.boxed_ty()
1082                && let Some(target_boxed) = target_ty.boxed_ty() =>
1083        {
1084            tcx.struct_lockstep_tails_for_codegen(source_boxed, target_boxed, typing_env)
1085        }
1086
1087        (&ty::Adt(source_adt_def, source_args), &ty::Adt(target_adt_def, target_args)) => {
1088            assert_eq!(source_adt_def, target_adt_def);
1089            let CustomCoerceUnsized::Struct(coerce_index) =
1090                match crate::custom_coerce_unsize_info(tcx, source_ty, target_ty) {
1091                    Ok(ccu) => ccu,
1092                    Err(e) => {
1093                        let e = Ty::new_error(tcx.tcx, e);
1094                        return (e, e);
1095                    }
1096                };
1097            let coerce_field = &source_adt_def.non_enum_variant().fields[coerce_index];
1098            // We're getting a possibly unnormalized type, so normalize it.
1099            let source_field =
1100                tcx.normalize_erasing_regions(typing_env, coerce_field.ty(*tcx, source_args));
1101            let target_field =
1102                tcx.normalize_erasing_regions(typing_env, coerce_field.ty(*tcx, target_args));
1103            find_tails_for_unsizing(tcx, source_field, target_field)
1104        }
1105
1106        _ => bug!(
1107            "find_vtable_types_for_unsizing: invalid coercion {:?} -> {:?}",
1108            source_ty,
1109            target_ty
1110        ),
1111    }
1112}
1113
1114#[instrument(skip(tcx), level = "debug", ret)]
1115fn create_fn_mono_item<'tcx>(
1116    tcx: TyCtxt<'tcx>,
1117    instance: Instance<'tcx>,
1118    source: Span,
1119) -> Spanned<MonoItem<'tcx>> {
1120    let def_id = instance.def_id();
1121    if tcx.sess.opts.unstable_opts.profile_closures
1122        && def_id.is_local()
1123        && tcx.is_closure_like(def_id)
1124    {
1125        crate::util::dump_closure_profile(tcx, instance);
1126    }
1127
1128    respan(source, MonoItem::Fn(instance))
1129}
1130
1131/// Creates a `MonoItem` for each method that is referenced by the vtable for
1132/// the given trait/impl pair.
1133fn create_mono_items_for_vtable_methods<'tcx>(
1134    tcx: TyCtxt<'tcx>,
1135    trait_ty: Ty<'tcx>,
1136    impl_ty: Ty<'tcx>,
1137    source: Span,
1138    output: &mut MonoItems<'tcx>,
1139) {
1140    assert!(!trait_ty.has_escaping_bound_vars() && !impl_ty.has_escaping_bound_vars());
1141
1142    let ty::Dynamic(trait_ty, ..) = trait_ty.kind() else {
1143        bug!("create_mono_items_for_vtable_methods: {trait_ty:?} not a trait type");
1144    };
1145    if let Some(principal) = trait_ty.principal() {
1146        let trait_ref =
1147            tcx.instantiate_bound_regions_with_erased(principal.with_self_ty(tcx, impl_ty));
1148        assert!(!trait_ref.has_escaping_bound_vars());
1149
1150        // Walk all methods of the trait, including those of its supertraits
1151        let entries = tcx.vtable_entries(trait_ref);
1152        debug!(?entries);
1153        let methods = entries
1154            .iter()
1155            .filter_map(|entry| match entry {
1156                VtblEntry::MetadataDropInPlace
1157                | VtblEntry::MetadataSize
1158                | VtblEntry::MetadataAlign
1159                | VtblEntry::Vacant => None,
1160                VtblEntry::TraitVPtr(_) => {
1161                    // all super trait items already covered, so skip them.
1162                    None
1163                }
1164                VtblEntry::Method(instance) => {
1165                    Some(*instance).filter(|instance| tcx.should_codegen_locally(*instance))
1166                }
1167            })
1168            .map(|item| create_fn_mono_item(tcx, item, source));
1169        output.extend(methods);
1170    }
1171
1172    // Also add the destructor, if it's necessary.
1173    //
1174    // This matches the check in vtable_allocation_provider in middle/ty/vtable.rs,
1175    // if we don't need drop we're not adding an actual pointer to the vtable.
1176    if impl_ty.needs_drop(tcx, ty::TypingEnv::fully_monomorphized()) {
1177        visit_drop_use(tcx, impl_ty, false, source, output);
1178    }
1179}
1180
1181/// Scans the CTFE alloc in order to find function pointers and statics that must be monomorphized.
1182fn collect_alloc<'tcx>(tcx: TyCtxt<'tcx>, alloc_id: AllocId, output: &mut MonoItems<'tcx>) {
1183    match tcx.global_alloc(alloc_id) {
1184        GlobalAlloc::Static(def_id) => {
1185            assert!(!tcx.is_thread_local_static(def_id));
1186            let instance = Instance::mono(tcx, def_id);
1187            if tcx.should_codegen_locally(instance) {
1188                trace!("collecting static {:?}", def_id);
1189                output.push(dummy_spanned(MonoItem::Static(def_id)));
1190            }
1191        }
1192        GlobalAlloc::Memory(alloc) => {
1193            trace!("collecting {:?} with {:#?}", alloc_id, alloc);
1194            let ptrs = alloc.inner().provenance().ptrs();
1195            // avoid `ensure_sufficient_stack` in the common case of "no pointers"
1196            if !ptrs.is_empty() {
1197                rustc_data_structures::stack::ensure_sufficient_stack(move || {
1198                    for &prov in ptrs.values() {
1199                        collect_alloc(tcx, prov.alloc_id(), output);
1200                    }
1201                });
1202            }
1203        }
1204        GlobalAlloc::Function { instance, .. } => {
1205            if tcx.should_codegen_locally(instance) {
1206                trace!("collecting {:?} with {:#?}", alloc_id, instance);
1207                output.push(create_fn_mono_item(tcx, instance, DUMMY_SP));
1208            }
1209        }
1210        GlobalAlloc::VTable(ty, dyn_ty) => {
1211            let alloc_id = tcx.vtable_allocation((
1212                ty,
1213                dyn_ty
1214                    .principal()
1215                    .map(|principal| tcx.instantiate_bound_regions_with_erased(principal)),
1216            ));
1217            collect_alloc(tcx, alloc_id, output)
1218        }
1219        GlobalAlloc::TypeId { .. } => {}
1220    }
1221}
1222
1223/// Scans the MIR in order to find function calls, closures, and drop-glue.
1224///
1225/// Anything that's found is added to `output`. Furthermore the "mentioned items" of the MIR are returned.
1226#[instrument(skip(tcx), level = "debug")]
1227fn collect_items_of_instance<'tcx>(
1228    tcx: TyCtxt<'tcx>,
1229    instance: Instance<'tcx>,
1230    mode: CollectionMode,
1231) -> (MonoItems<'tcx>, MonoItems<'tcx>) {
1232    // This item is getting monomorphized, do mono-time checks.
1233    tcx.ensure_ok().check_mono_item(instance);
1234
1235    let body = tcx.instance_mir(instance.def);
1236    // Naively, in "used" collection mode, all functions get added to *both* `used_items` and
1237    // `mentioned_items`. Mentioned items processing will then notice that they have already been
1238    // visited, but at that point each mentioned item has been monomorphized, added to the
1239    // `mentioned_items` worklist, and checked in the global set of visited items. To remove that
1240    // overhead, we have a special optimization that avoids adding items to `mentioned_items` when
1241    // they are already added in `used_items`. We could just scan `used_items`, but that's a linear
1242    // scan and not very efficient. Furthermore we can only do that *after* monomorphizing the
1243    // mentioned item. So instead we collect all pre-monomorphized `MentionedItem` that were already
1244    // added to `used_items` in a hash set, which can efficiently query in the
1245    // `body.mentioned_items` loop below without even having to monomorphize the item.
1246    let mut used_items = MonoItems::new();
1247    let mut mentioned_items = MonoItems::new();
1248    let mut used_mentioned_items = Default::default();
1249    let mut collector = MirUsedCollector {
1250        tcx,
1251        body,
1252        used_items: &mut used_items,
1253        used_mentioned_items: &mut used_mentioned_items,
1254        instance,
1255    };
1256
1257    if mode == CollectionMode::UsedItems {
1258        if tcx.sess.opts.debuginfo == DebugInfo::Full {
1259            for var_debug_info in &body.var_debug_info {
1260                collector.visit_var_debug_info(var_debug_info);
1261            }
1262        }
1263        for (bb, data) in traversal::mono_reachable(body, tcx, instance) {
1264            collector.visit_basic_block_data(bb, data)
1265        }
1266    }
1267
1268    // Always visit all `required_consts`, so that we evaluate them and abort compilation if any of
1269    // them errors.
1270    for const_op in body.required_consts() {
1271        if let Some(val) = collector.eval_constant(const_op) {
1272            collect_const_value(tcx, val, &mut mentioned_items);
1273        }
1274    }
1275
1276    // Always gather mentioned items. We try to avoid processing items that we have already added to
1277    // `used_items` above.
1278    for item in body.mentioned_items() {
1279        if !collector.used_mentioned_items.contains(&item.node) {
1280            let item_mono = collector.monomorphize(item.node);
1281            visit_mentioned_item(tcx, &item_mono, item.span, &mut mentioned_items);
1282        }
1283    }
1284
1285    (used_items, mentioned_items)
1286}
1287
1288fn items_of_instance<'tcx>(
1289    tcx: TyCtxt<'tcx>,
1290    (instance, mode): (Instance<'tcx>, CollectionMode),
1291) -> (&'tcx [Spanned<MonoItem<'tcx>>], &'tcx [Spanned<MonoItem<'tcx>>]) {
1292    let (used_items, mentioned_items) = collect_items_of_instance(tcx, instance, mode);
1293
1294    let used_items = tcx.arena.alloc_from_iter(used_items);
1295    let mentioned_items = tcx.arena.alloc_from_iter(mentioned_items);
1296
1297    (used_items, mentioned_items)
1298}
1299
1300/// `item` must be already monomorphized.
1301#[instrument(skip(tcx, span, output), level = "debug")]
1302fn visit_mentioned_item<'tcx>(
1303    tcx: TyCtxt<'tcx>,
1304    item: &MentionedItem<'tcx>,
1305    span: Span,
1306    output: &mut MonoItems<'tcx>,
1307) {
1308    match *item {
1309        MentionedItem::Fn(ty) => {
1310            if let ty::FnDef(def_id, args) = *ty.kind() {
1311                let instance = Instance::expect_resolve(
1312                    tcx,
1313                    ty::TypingEnv::fully_monomorphized(),
1314                    def_id,
1315                    args,
1316                    span,
1317                );
1318                // `visit_instance_use` was written for "used" item collection but works just as well
1319                // for "mentioned" item collection.
1320                // We can set `is_direct_call`; that just means we'll skip a bunch of shims that anyway
1321                // can't have their own failing constants.
1322                visit_instance_use(tcx, instance, /*is_direct_call*/ true, span, output);
1323            }
1324        }
1325        MentionedItem::Drop(ty) => {
1326            visit_drop_use(tcx, ty, /*is_direct_call*/ true, span, output);
1327        }
1328        MentionedItem::UnsizeCast { source_ty, target_ty } => {
1329            let (source_ty, target_ty) =
1330                find_tails_for_unsizing(tcx.at(span), source_ty, target_ty);
1331            // This could also be a different Unsize instruction, like
1332            // from a fixed sized array to a slice. But we are only
1333            // interested in things that produce a vtable.
1334            if target_ty.is_trait() && !source_ty.is_trait() {
1335                create_mono_items_for_vtable_methods(tcx, target_ty, source_ty, span, output);
1336            }
1337        }
1338        MentionedItem::Closure(source_ty) => {
1339            if let ty::Closure(def_id, args) = *source_ty.kind() {
1340                let instance =
1341                    Instance::resolve_closure(tcx, def_id, args, ty::ClosureKind::FnOnce);
1342                if tcx.should_codegen_locally(instance) {
1343                    output.push(create_fn_mono_item(tcx, instance, span));
1344                }
1345            } else {
1346                bug!()
1347            }
1348        }
1349    }
1350}
1351
1352#[instrument(skip(tcx, output), level = "debug")]
1353fn collect_const_value<'tcx>(
1354    tcx: TyCtxt<'tcx>,
1355    value: mir::ConstValue,
1356    output: &mut MonoItems<'tcx>,
1357) {
1358    match value {
1359        mir::ConstValue::Scalar(Scalar::Ptr(ptr, _size)) => {
1360            collect_alloc(tcx, ptr.provenance.alloc_id(), output)
1361        }
1362        mir::ConstValue::Indirect { alloc_id, .. }
1363        | mir::ConstValue::Slice { alloc_id, meta: _ } => collect_alloc(tcx, alloc_id, output),
1364        _ => {}
1365    }
1366}
1367
1368//=-----------------------------------------------------------------------------
1369// Root Collection
1370//=-----------------------------------------------------------------------------
1371
1372// Find all non-generic items by walking the HIR. These items serve as roots to
1373// start monomorphizing from.
1374#[instrument(skip(tcx, mode), level = "debug")]
1375fn collect_roots(tcx: TyCtxt<'_>, mode: MonoItemCollectionStrategy) -> Vec<MonoItem<'_>> {
1376    debug!("collecting roots");
1377    let mut roots = MonoItems::new();
1378
1379    {
1380        let entry_fn = tcx.entry_fn(());
1381
1382        debug!("collect_roots: entry_fn = {:?}", entry_fn);
1383
1384        let mut collector = RootCollector { tcx, strategy: mode, entry_fn, output: &mut roots };
1385
1386        let crate_items = tcx.hir_crate_items(());
1387
1388        for id in crate_items.free_items() {
1389            collector.process_item(id);
1390        }
1391
1392        for id in crate_items.impl_items() {
1393            collector.process_impl_item(id);
1394        }
1395
1396        for id in crate_items.nested_bodies() {
1397            collector.process_nested_body(id);
1398        }
1399
1400        collector.push_extra_entry_roots();
1401    }
1402
1403    // We can only codegen items that are instantiable - items all of
1404    // whose predicates hold. Luckily, items that aren't instantiable
1405    // can't actually be used, so we can just skip codegenning them.
1406    roots
1407        .into_iter()
1408        .filter_map(|Spanned { node: mono_item, .. }| {
1409            mono_item.is_instantiable(tcx).then_some(mono_item)
1410        })
1411        .collect()
1412}
1413
1414struct RootCollector<'a, 'tcx> {
1415    tcx: TyCtxt<'tcx>,
1416    strategy: MonoItemCollectionStrategy,
1417    output: &'a mut MonoItems<'tcx>,
1418    entry_fn: Option<(DefId, EntryFnType)>,
1419}
1420
1421impl<'v> RootCollector<'_, 'v> {
1422    fn process_item(&mut self, id: hir::ItemId) {
1423        match self.tcx.def_kind(id.owner_id) {
1424            DefKind::Enum | DefKind::Struct | DefKind::Union => {
1425                if self.strategy == MonoItemCollectionStrategy::Eager
1426                    && !self.tcx.generics_of(id.owner_id).requires_monomorphization(self.tcx)
1427                {
1428                    debug!("RootCollector: ADT drop-glue for `{id:?}`",);
1429                    let id_args =
1430                        ty::GenericArgs::for_item(self.tcx, id.owner_id.to_def_id(), |param, _| {
1431                            match param.kind {
1432                                GenericParamDefKind::Lifetime => {
1433                                    self.tcx.lifetimes.re_erased.into()
1434                                }
1435                                GenericParamDefKind::Type { .. }
1436                                | GenericParamDefKind::Const { .. } => {
1437                                    unreachable!(
1438                                        "`own_requires_monomorphization` check means that \
1439                                we should have no type/const params"
1440                                    )
1441                                }
1442                            }
1443                        });
1444
1445                    // This type is impossible to instantiate, so we should not try to
1446                    // generate a `drop_in_place` instance for it.
1447                    if self.tcx.instantiate_and_check_impossible_predicates((
1448                        id.owner_id.to_def_id(),
1449                        id_args,
1450                    )) {
1451                        return;
1452                    }
1453
1454                    let ty =
1455                        self.tcx.type_of(id.owner_id.to_def_id()).instantiate(self.tcx, id_args);
1456                    assert!(!ty.has_non_region_param());
1457                    visit_drop_use(self.tcx, ty, true, DUMMY_SP, self.output);
1458                }
1459            }
1460            DefKind::GlobalAsm => {
1461                debug!(
1462                    "RootCollector: ItemKind::GlobalAsm({})",
1463                    self.tcx.def_path_str(id.owner_id)
1464                );
1465                self.output.push(dummy_spanned(MonoItem::GlobalAsm(id)));
1466            }
1467            DefKind::Static { .. } => {
1468                let def_id = id.owner_id.to_def_id();
1469                debug!("RootCollector: ItemKind::Static({})", self.tcx.def_path_str(def_id));
1470                self.output.push(dummy_spanned(MonoItem::Static(def_id)));
1471            }
1472            DefKind::Const => {
1473                // Const items only generate mono items if they are actually used somewhere.
1474                // Just declaring them is insufficient.
1475
1476                // If we're collecting items eagerly, then recurse into all constants.
1477                // Otherwise the value is only collected when explicitly mentioned in other items.
1478                if self.strategy == MonoItemCollectionStrategy::Eager {
1479                    if !self.tcx.generics_of(id.owner_id).own_requires_monomorphization()
1480                        && let Ok(val) = self.tcx.const_eval_poly(id.owner_id.to_def_id())
1481                    {
1482                        collect_const_value(self.tcx, val, self.output);
1483                    }
1484                }
1485            }
1486            DefKind::Impl { .. } => {
1487                if self.strategy == MonoItemCollectionStrategy::Eager {
1488                    create_mono_items_for_default_impls(self.tcx, id, self.output);
1489                }
1490            }
1491            DefKind::Fn => {
1492                self.push_if_root(id.owner_id.def_id);
1493            }
1494            _ => {}
1495        }
1496    }
1497
1498    fn process_impl_item(&mut self, id: hir::ImplItemId) {
1499        if matches!(self.tcx.def_kind(id.owner_id), DefKind::AssocFn) {
1500            self.push_if_root(id.owner_id.def_id);
1501        }
1502    }
1503
1504    fn process_nested_body(&mut self, def_id: LocalDefId) {
1505        match self.tcx.def_kind(def_id) {
1506            DefKind::Closure => {
1507                if self.strategy == MonoItemCollectionStrategy::Eager
1508                    && !self
1509                        .tcx
1510                        .generics_of(self.tcx.typeck_root_def_id(def_id.to_def_id()))
1511                        .requires_monomorphization(self.tcx)
1512                {
1513                    let instance = match *self.tcx.type_of(def_id).instantiate_identity().kind() {
1514                        ty::Closure(def_id, args)
1515                        | ty::Coroutine(def_id, args)
1516                        | ty::CoroutineClosure(def_id, args) => {
1517                            Instance::new_raw(def_id, self.tcx.erase_regions(args))
1518                        }
1519                        _ => unreachable!(),
1520                    };
1521                    let Ok(instance) = self.tcx.try_normalize_erasing_regions(
1522                        ty::TypingEnv::fully_monomorphized(),
1523                        instance,
1524                    ) else {
1525                        // Don't ICE on an impossible-to-normalize closure.
1526                        return;
1527                    };
1528                    let mono_item = create_fn_mono_item(self.tcx, instance, DUMMY_SP);
1529                    if mono_item.node.is_instantiable(self.tcx) {
1530                        self.output.push(mono_item);
1531                    }
1532                }
1533            }
1534            _ => {}
1535        }
1536    }
1537
1538    fn is_root(&self, def_id: LocalDefId) -> bool {
1539        !self.tcx.generics_of(def_id).requires_monomorphization(self.tcx)
1540            && match self.strategy {
1541                MonoItemCollectionStrategy::Eager => {
1542                    !matches!(self.tcx.codegen_fn_attrs(def_id).inline, InlineAttr::Force { .. })
1543                }
1544                MonoItemCollectionStrategy::Lazy => {
1545                    self.entry_fn.and_then(|(id, _)| id.as_local()) == Some(def_id)
1546                        || self.tcx.is_reachable_non_generic(def_id)
1547                        || self
1548                            .tcx
1549                            .codegen_fn_attrs(def_id)
1550                            .flags
1551                            .contains(CodegenFnAttrFlags::RUSTC_STD_INTERNAL_SYMBOL)
1552                }
1553            }
1554    }
1555
1556    /// If `def_id` represents a root, pushes it onto the list of
1557    /// outputs. (Note that all roots must be monomorphic.)
1558    #[instrument(skip(self), level = "debug")]
1559    fn push_if_root(&mut self, def_id: LocalDefId) {
1560        if self.is_root(def_id) {
1561            debug!("found root");
1562
1563            let instance = Instance::mono(self.tcx, def_id.to_def_id());
1564            self.output.push(create_fn_mono_item(self.tcx, instance, DUMMY_SP));
1565        }
1566    }
1567
1568    /// As a special case, when/if we encounter the
1569    /// `main()` function, we also have to generate a
1570    /// monomorphized copy of the start lang item based on
1571    /// the return type of `main`. This is not needed when
1572    /// the user writes their own `start` manually.
1573    fn push_extra_entry_roots(&mut self) {
1574        let Some((main_def_id, EntryFnType::Main { .. })) = self.entry_fn else {
1575            return;
1576        };
1577
1578        let main_instance = Instance::mono(self.tcx, main_def_id);
1579        if self.tcx.should_codegen_locally(main_instance) {
1580            self.output.push(create_fn_mono_item(
1581                self.tcx,
1582                main_instance,
1583                self.tcx.def_span(main_def_id),
1584            ));
1585        }
1586
1587        let Some(start_def_id) = self.tcx.lang_items().start_fn() else {
1588            self.tcx.dcx().emit_fatal(errors::StartNotFound);
1589        };
1590        let main_ret_ty = self.tcx.fn_sig(main_def_id).no_bound_vars().unwrap().output();
1591
1592        // Given that `main()` has no arguments,
1593        // then its return type cannot have
1594        // late-bound regions, since late-bound
1595        // regions must appear in the argument
1596        // listing.
1597        let main_ret_ty = self.tcx.normalize_erasing_regions(
1598            ty::TypingEnv::fully_monomorphized(),
1599            main_ret_ty.no_bound_vars().unwrap(),
1600        );
1601
1602        let start_instance = Instance::expect_resolve(
1603            self.tcx,
1604            ty::TypingEnv::fully_monomorphized(),
1605            start_def_id,
1606            self.tcx.mk_args(&[main_ret_ty.into()]),
1607            DUMMY_SP,
1608        );
1609
1610        self.output.push(create_fn_mono_item(self.tcx, start_instance, DUMMY_SP));
1611    }
1612}
1613
1614#[instrument(level = "debug", skip(tcx, output))]
1615fn create_mono_items_for_default_impls<'tcx>(
1616    tcx: TyCtxt<'tcx>,
1617    item: hir::ItemId,
1618    output: &mut MonoItems<'tcx>,
1619) {
1620    let Some(impl_) = tcx.impl_trait_header(item.owner_id) else {
1621        return;
1622    };
1623
1624    if matches!(impl_.polarity, ty::ImplPolarity::Negative) {
1625        return;
1626    }
1627
1628    if tcx.generics_of(item.owner_id).own_requires_monomorphization() {
1629        return;
1630    }
1631
1632    // Lifetimes never affect trait selection, so we are allowed to eagerly
1633    // instantiate an instance of an impl method if the impl (and method,
1634    // which we check below) is only parameterized over lifetime. In that case,
1635    // we use the ReErased, which has no lifetime information associated with
1636    // it, to validate whether or not the impl is legal to instantiate at all.
1637    let only_region_params = |param: &ty::GenericParamDef, _: &_| match param.kind {
1638        GenericParamDefKind::Lifetime => tcx.lifetimes.re_erased.into(),
1639        GenericParamDefKind::Type { .. } | GenericParamDefKind::Const { .. } => {
1640            unreachable!(
1641                "`own_requires_monomorphization` check means that \
1642                we should have no type/const params"
1643            )
1644        }
1645    };
1646    let impl_args = GenericArgs::for_item(tcx, item.owner_id.to_def_id(), only_region_params);
1647    let trait_ref = impl_.trait_ref.instantiate(tcx, impl_args);
1648
1649    // Unlike 'lazy' monomorphization that begins by collecting items transitively
1650    // called by `main` or other global items, when eagerly monomorphizing impl
1651    // items, we never actually check that the predicates of this impl are satisfied
1652    // in a empty param env (i.e. with no assumptions).
1653    //
1654    // Even though this impl has no type or const generic parameters, because we don't
1655    // consider higher-ranked predicates such as `for<'a> &'a mut [u8]: Copy` to
1656    // be trivially false. We must now check that the impl has no impossible-to-satisfy
1657    // predicates.
1658    if tcx.instantiate_and_check_impossible_predicates((item.owner_id.to_def_id(), impl_args)) {
1659        return;
1660    }
1661
1662    let typing_env = ty::TypingEnv::fully_monomorphized();
1663    let trait_ref = tcx.normalize_erasing_regions(typing_env, trait_ref);
1664    let overridden_methods = tcx.impl_item_implementor_ids(item.owner_id);
1665    for method in tcx.provided_trait_methods(trait_ref.def_id) {
1666        if overridden_methods.contains_key(&method.def_id) {
1667            continue;
1668        }
1669
1670        if tcx.generics_of(method.def_id).own_requires_monomorphization() {
1671            continue;
1672        }
1673
1674        // As mentioned above, the method is legal to eagerly instantiate if it
1675        // only has lifetime generic parameters. This is validated by calling
1676        // `own_requires_monomorphization` on both the impl and method.
1677        let args = trait_ref.args.extend_to(tcx, method.def_id, only_region_params);
1678        let instance = ty::Instance::expect_resolve(tcx, typing_env, method.def_id, args, DUMMY_SP);
1679
1680        let mono_item = create_fn_mono_item(tcx, instance, DUMMY_SP);
1681        if mono_item.node.is_instantiable(tcx) && tcx.should_codegen_locally(instance) {
1682            output.push(mono_item);
1683        }
1684    }
1685}
1686
1687//=-----------------------------------------------------------------------------
1688// Top-level entry point, tying it all together
1689//=-----------------------------------------------------------------------------
1690
1691#[instrument(skip(tcx, strategy), level = "debug")]
1692pub(crate) fn collect_crate_mono_items<'tcx>(
1693    tcx: TyCtxt<'tcx>,
1694    strategy: MonoItemCollectionStrategy,
1695) -> (Vec<MonoItem<'tcx>>, UsageMap<'tcx>) {
1696    let _prof_timer = tcx.prof.generic_activity("monomorphization_collector");
1697
1698    let roots = tcx
1699        .sess
1700        .time("monomorphization_collector_root_collections", || collect_roots(tcx, strategy));
1701
1702    debug!("building mono item graph, beginning at roots");
1703
1704    let state = SharedState {
1705        visited: MTLock::new(UnordSet::default()),
1706        mentioned: MTLock::new(UnordSet::default()),
1707        usage_map: MTLock::new(UsageMap::new()),
1708    };
1709    let recursion_limit = tcx.recursion_limit();
1710
1711    tcx.sess.time("monomorphization_collector_graph_walk", || {
1712        par_for_each_in(roots, |root| {
1713            collect_items_root(tcx, dummy_spanned(*root), &state, recursion_limit);
1714        });
1715    });
1716
1717    // The set of MonoItems was created in an inherently indeterministic order because
1718    // of parallelism. We sort it here to ensure that the output is deterministic.
1719    let mono_items = tcx.with_stable_hashing_context(move |ref hcx| {
1720        state.visited.into_inner().into_sorted(hcx, true)
1721    });
1722
1723    (mono_items, state.usage_map.into_inner())
1724}
1725
1726pub(crate) fn provide(providers: &mut Providers) {
1727    providers.hooks.should_codegen_locally = should_codegen_locally;
1728    providers.items_of_instance = items_of_instance;
1729}