Sporadic panic compiling with low available resources

[gtema]

Sadly I can't present a code, since it happens during compiling of a pretty big project on a system with low resources (2cpu). The error is not reproducible and happens usually when system is under pressure.

Meta

rustc --version --verbose:

rustc 1.78.0 (9b00956e5 2024-04-29)
binary: rustc
commit-hash: 9b00956e56009bab2aa15d7bff10916599e3d6d6
commit-date: 2024-04-29
host: x86_64-unknown-linux-gnu
release: 1.78.0
LLVM version: 18.1.2

sccache is also being used.

Backtrace

thread 'rustc' panicked at compiler/rustc_metadata/src/rmeta/decoder.rs:456:36:
Missing SyntaxContext 1802102 for crate "openstack_cli"
stack backtrace:
stack backtrace:
   0:     0x7f228b0b4892 - std::backtrace_rs::backtrace::libunwind::trace::he4ee80166a02c846
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/../../backtrace/src/backtrace/libunwind.rs:105:5
   1:     0x7f228b0b4892 - std::backtrace_rs::backtrace::trace_unsynchronized::h476faccf57e88641
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7f228b0b4892 - std::sys_common::backtrace::_print_fmt::h430c922a77e7a59c
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:68:5
   3:     0x7f228b0b4892 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hffecb437d922f988
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x7f228b105a6c - core::fmt::rt::Argument::fmt::hf3df69369399bfa9
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/fmt/rt.rs:142:9
   5:     0x7f228b105a6c - core::fmt::write::hd9a8d7d029f9ea1a
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/fmt/mod.rs:1153:17
   6:     0x7f228b0a978f - std::io::Write::write_fmt::h0e1226b2b8d973fe
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/io/mod.rs:1843:15
   7:     0x7f228b0b4664 - std::sys_common::backtrace::_print::hd2df4a083f6e69b8
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x7f228b0b4664 - std::sys_common::backtrace::print::he907f6ad7eee41cb
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x7f228b0b735b - std::panicking::default_hook::{{closure}}::h3926193b61c9ca9b
  10:     0x7f228b0b70b3 - std::panicking::default_hook::h25ba2457dea68e65
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:292:9
  11:     0x7f2287e2845d - std[e4dfbc2c3f4b09f1]::panicking::update_hook::<alloc[1adba907b9db1888]::boxed::Box<rustc_driver_impl[24a943716c49befe]::install_ice_hook::{closure#0}>>::{closure#0}
  12:     0x7f228b0b7ac0 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h022ca2c0d8c21c9e
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2034:9
  13:     0x7f228b0b7ac0 - std::panicking::rust_panic_with_hook::h0ad14d90dcf5224f
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:783:13
  14:     0x7f228b0b7802 - std::panicking::begin_panic_handler::{{closure}}::h4a1838a06f542647
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:657:13
  15:     0x7f228b0b4d66 - std::sys_common::backtrace::__rust_end_short_backtrace::h77cc4dc3567ca904
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:171:18
  16:     0x7f228b0b7534 - rust_begin_unwind
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5
  17:     0x7f228b101f85 - core::panicking::panic_fmt::h940d4fd01a4b4fd1
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14
  18:     0x7f22891cd967 - <rustc_metadata[70192afe97828208]::rmeta::decoder::DecodeContext as rustc_span[a4517f2b2e65298c]::SpanDecoder>::decode_syntax_context
  19:     0x7f22891ceef1 - <rustc_metadata[70192afe97828208]::rmeta::decoder::DecodeContext as rustc_span[a4517f2b2e65298c]::SpanDecoder>::decode_span
  20:     0x7f22891c67fb - <rustc_middle[fda44fdb505d3e7f]::mir::Body as rustc_serialize[8411edc67acff09a]::serialize::Decodable<rustc_metadata[70192afe97828208]::rmeta::decoder::DecodeContext>>::decode
  21:     0x7f228970423d - rustc_metadata[70192afe97828208]::rmeta::decoder::cstore_impl::provide_extern::optimized_mir
  22:     0x7f2289703fad - rustc_query_impl[e4152ad88c3d6c78]::plumbing::__rust_begin_short_backtrace::<rustc_query_impl[e4152ad88c3d6c78]::query_impl::optimized_mir::dynamic_query::{closure#2}::{closure#0}, rustc_middle[fda44fdb505d3e7f]::query::erase::Erased<[u8; 8usize]>>
  23:     0x7f2289394c99 - rustc_query_system[475239fef39bf53f]::query::plumbing::try_execute_query::<rustc_query_impl[e4152ad88c3d6c78]::DynamicConfig<rustc_query_system[475239fef39bf53f]::query::caches::DefIdCache<rustc_middle[fda44fdb505d3e7f]::query::erase::Erased<[u8; 8usize]>>, false, false, false>, rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt, true>
  24:     0x7f2289c46f59 - rustc_query_impl[e4152ad88c3d6c78]::plumbing::force_from_dep_node::<rustc_query_impl[e4152ad88c3d6c78]::DynamicConfig<rustc_query_system[475239fef39bf53f]::query::caches::DefIdCache<rustc_middle[fda44fdb505d3e7f]::query::erase::Erased<[u8; 8usize]>>, false, false, false>>
  25:     0x7f22885cafcd - <rustc_query_impl[e4152ad88c3d6c78]::plumbing::query_callback<rustc_query_impl[e4152ad88c3d6c78]::query_impl::optimized_mir::QueryType>::{closure#0} as core[836963c7c1decc11]::ops::function::FnOnce<(rustc_middle[fda44fdb505d3e7f]::ty::context::TyCtxt, rustc_query_system[475239fef39bf53f]::dep_graph::dep_node::DepNode)>>::call_once
  26:     0x7f2289029055 - <rustc_query_system[475239fef39bf53f]::dep_graph::graph::DepGraphData<rustc_middle[fda44fdb505d3e7f]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  27:     0x7f2289028fbd - <rustc_query_system[475239fef39bf53f]::dep_graph::graph::DepGraphData<rustc_middle[fda44fdb505d3e7f]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  28:     0x7f2289028fbd - <rustc_query_system[475239fef39bf53f]::dep_graph::graph::DepGraphData<rustc_middle[fda44fdb505d3e7f]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  29:     0x7f2289028fbd - <rustc_query_system[475239fef39bf53f]::dep_graph::graph::DepGraphData<rustc_middle[fda44fdb505d3e7f]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  30:     0x7f2289028fbd - <rustc_query_system[475239fef39bf53f]::dep_graph::graph::DepGraphData<rustc_middle[fda44fdb505d3e7f]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  31:     0x7f2289028fbd - <rustc_query_system[475239fef39bf53f]::dep_graph::graph::DepGraphData<rustc_middle[fda44fdb505d3e7f]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  32:     0x7f2289028fbd - <rustc_query_system[475239fef39bf53f]::dep_graph::graph::DepGraphData<rustc_middle[fda44fdb505d3e7f]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  33:     0x7f2289028fbd - <rustc_query_system[475239fef39bf53f]::dep_graph::graph::DepGraphData<rustc_middle[fda44fdb505d3e7f]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  34:     0x7f2289028fbd - <rustc_query_system[475239fef39bf53f]::dep_graph::graph::DepGraphData<rustc_middle[fda44fdb505d3e7f]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  35:     0x7f2289028fbd - <rustc_query_system[475239fef39bf53f]::dep_graph::graph::DepGraphData<rustc_middle[fda44fdb505d3e7f]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  36:     0x7f2289028fbd - <rustc_query_system[475239fef39bf53f]::dep_graph::graph::DepGraphData<rustc_middle[fda44fdb505d3e7f]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  37:     0x7f2289028fbd - <rustc_query_system[475239fef39bf53f]::dep_graph::graph::DepGraphData<rustc_middle[fda44fdb505d3e7f]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  38:     0x7f2289028fbd - <rustc_query_system[475239fef39bf53f]::dep_graph::graph::DepGraphData<rustc_middle[fda44fdb505d3e7f]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  39:     0x7f2289028fbd - <rustc_query_system[475239fef39bf53f]::dep_graph::graph::DepGraphData<rustc_middle[fda44fdb505d3e7f]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  40:     0x7f2289028fbd - <rustc_query_system[475239fef39bf53f]::dep_graph::graph::DepGraphData<rustc_middle[fda44fdb505d3e7f]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  41:     0x7f2289028fbd - <rustc_query_system[475239fef39bf53f]::dep_graph::graph::DepGraphData<rustc_middle[fda44fdb505d3e7f]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  42:     0x7f2289028fbd - <rustc_query_system[475239fef39bf53f]::dep_graph::graph::DepGraphData<rustc_middle[fda44fdb505d3e7f]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  43:     0x7f2289390b19 - rustc_query_system[475239fef39bf53f]::query::plumbing::ensure_must_run::<rustc_query_impl[e4152ad88c3d6c78]::DynamicConfig<rustc_query_system[475239fef39bf53f]::query::caches::VecCache<rustc_hir[33aca5c26ba91407]::hir_id::OwnerId, rustc_middle[fda44fdb505d3e7f]::query::erase::Erased<[u8; 1usize]>>, false, false, false>, rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt>
  44:     0x7f2289390cb7 - rustc_query_impl[e4152ad88c3d6c78]::query_impl::mir_drops_elaborated_and_const_checked::get_query_incr::__rust_end_short_backtrace
  45:     0x7f2289912284 - rustc_interface[ba2b6dc4c96cb491]::passes::analysis
  46:     0x7f22899118e5 - rustc_query_impl[e4152ad88c3d6c78]::plumbing::__rust_begin_short_backtrace::<rustc_query_impl[e4152ad88c3d6c78]::query_impl::analysis::dynamic_query::{closure#2}::{closure#0}, rustc_middle[fda44fdb505d3e7f]::query::erase::Erased<[u8; 1usize]>>
  47:     0x7f2289e9d375 - rustc_query_system[475239fef39bf53f]::query::plumbing::try_execute_query::<rustc_query_impl[e4152ad88c3d6c78]::DynamicConfig<rustc_query_system[475239fef39bf53f]::query::caches::SingleCache<rustc_middle[fda44fdb505d3e7f]::query::erase::Erased<[u8; 1usize]>>, false, false, false>, rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt, true>
  48:     0x7f2289e9cf9c - rustc_query_impl[e4152ad88c3d6c78]::query_impl::analysis::get_query_incr::__rust_end_short_backtrace
  49:     0x7f2289bfc814 - rustc_interface[ba2b6dc4c96cb491]::interface::run_compiler::<core[836963c7c1decc11]::result::Result<(), rustc_span[a4517f2b2e65298c]::ErrorGuaranteed>, rustc_driver_impl[24a943716c49befe]::run_compiler::{closure#0}>::{closure#0}
  50:     0x7f2289f3eb6e - std[e4dfbc2c3f4b09f1]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[ba2b6dc4c96cb491]::util::run_in_thread_with_globals<rustc_interface[ba2b6dc4c96cb491]::interface::run_compiler<core[836963c7c1decc11]::result::Result<(), rustc_span[a4517f2b2e65298c]::ErrorGuaranteed>, rustc_driver_impl[24a943716c49befe]::run_compiler::{closure#0}>::{closure#0}, core[836963c7c1decc11]::result::Result<(), rustc_span[a4517f2b2e65298c]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[836963c7c1decc11]::result::Result<(), rustc_span[a4517f2b2e65298c]::ErrorGuaranteed>>
  51:     0x7f2289f3e9ca - <<std[e4dfbc2c3f4b09f1]::thread::Builder>::spawn_unchecked_<rustc_interface[ba2b6dc4c96cb491]::util::run_in_thread_with_globals<rustc_interface[ba2b6dc4c96cb491]::interface::run_compiler<core[836963c7c1decc11]::result::Result<(), rustc_span[a4517f2b2e65298c]::ErrorGuaranteed>, rustc_driver_impl[24a943716c49befe]::run_compiler::{closure#0}>::{closure#0}, core[836963c7c1decc11]::result::Result<(), rustc_span[a4517f2b2e65298c]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[836963c7c1decc11]::result::Result<(), rustc_span[a4517f2b2e65298c]::ErrorGuaranteed>>::{closure#1} as core[836963c7c1decc11]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  52:     0x7f228b0c1145 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h19b9e642d37e7272
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2020:9
  53:     0x7f228b0c1145 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h97265befc434d3ae
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2020:9
  54:     0x7f228b0c1145 - std::sys::pal::unix::thread::Thread::new::thread_start::h420dad5cf01a9f35
                               at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys/pal/unix/thread.rs:108:17
  55:     0x7f2284ea91b7 - start_thread
  56:     0x7f2284f2b39c - __GI___clone3
  57:                0x0 - <unknown>

note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.78.0 (9b00956e5 2024-04-29) running on x86_64-unknown-linux-gnu

note: compiler flags: --crate-type bin -C embed-bitcode=no -C incremental=[REDACTED] -C strip=debuginfo

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [optimized_mir] optimizing MIR for `openstack_cli::block_storage::v3::default_type::delete::DefaultTypeCommand::take_action::{closure#0}`
#1 [analysis] running analysis passes on this crate
end of query stack
there was a panic while trying to force a dep node
try_mark_green dep node stack:
#0 layout_of(50ce6e0430ea69d8-317261177dcface1)
#1 layout_of(4b1552e6bd747bc7-77ed0138a8d8d645)
#2 layout_of(4cf13c39e951339c-f5f82044666e223f)
#3 layout_of(a3dee3b30cf808c3-3a6e021221241706)
#4 layout_of(4f91719f6ca70abc-493fafac069970d0)
#5 layout_of(3a384e266c18be90-4b0fe244ec915ac2)
#6 layout_of(48ac620664a682ba-c326f5cce3af088)
#7 layout_of(c18751150576cec9-e8b4b127eac2f2e7)
#8 layout_of(234adbd073a917ac-5c716200772537f2)
#9 layout_of(706049fc7405dae3-3b7d1f7e90c1813c)
#10 layout_of(b81958a840e0f712-671f505983d84ea9)
#11 layout_of(e3beb139c38be6b4-f1976512ea2c9908)
#12 layout_of(5d504ab8a3fd0739-19489422ee55769e)
#13 layout_of(b2bb3ff529cfa3e0-747f09f5629d4e4a)
#14 layout_of(76aebd48c2d3fdb8-17f748b1343b4bbe)
#15 layout_of(b5b33ee778414687-f240d2dfa2fe6a1a)
#16 mir_drops_elaborated_and_const_checked(osc[a3b8]::main)
end of try_mark_green dep node stack
error: could not compile `openstack_cli` (bin "osc")

[workingjubilee]

@gtema Can you be more precise about resources? Drive space? RAM?

[workingjubilee]

I ask because the issue you mention is in the frontend, and as I understand it, the parallel frontend is not actually used by default (it is technically active but in single-threaded mode).

[gtema]

16G RAM and around 70Gb free disk (which is full in few weeks of project development, 2cpu (older i5)

[workingjubilee]

Hmmm, that's a bit low on the disk side but not so much that I would expect compiling to simply fail like that. I don't think the resources available are the problem, based on your description, as much weaker CPUs, with less RAM, and less disk, successfully compile projects.

Do you have to use ccache? The Rust compiler has its own incremental logic, and the panic here is firmly inside the "restore from previously-cached incremental compilation metadata" logic.

[gtema]

Scache is not mandatory, was just hoping to speed compilation up (since it takes around 5-7 minutes). I was suspecting the issue may lay in cache itself, since usually cargo clean helps to solve the situation for few days.

[workingjubilee]

Oh, sccache?

[gtema]

Yeah, autocorrect fixed it (facepalm) it's sccache