Compiled model methods and `expose_functions` fail to load when RcppParallel's TBB is already in the session

[jgabry]

I noticed this while working on v1.0 and also when reviewing #1274. I asked Claude to write up the report below.

---

init_model_methods() and expose_functions() compile the model methods with Rcpp and load the result. If RcppParallel's TBB is already loaded in the R process, which happens as soon as rstan or brms is loaded, the load fails:

Error in dyn.load(".../sourceCpp_2.so") :
  unable to load shared object '.../sourceCpp_2.so':
  dlopen(...): Symbol not found: __ZN3tbb8internal23allocate_via_handler_v3Em
  Expected in: /Users/.../R/x86_64/4.5/library/RcppParallel/lib/libtbb.dylib

The other order is no better. With the model methods built first, loadNamespace("RcppParallel") fails inside RcppParallel's .onLoad with a oneTBB symbol expected in CmdStan's libtbb.dylib, so rstan and brms can't be loaded in that session at all.

Without RcppParallel in the session the same calls succeed. Reproduced on macOS with R 4.5, RcppParallel 6.2.0, and CmdStan 2.39.0 and 2.40.0-rc1, on cmdstanr master and on the v1.0 branch.

Why

Stan Math's autodiff core includes TBB headers, so the compiled methods reference symbols from the TBB CmdStan bundles, which has been 2020.3 in every release from 2.27 on. RcppParallel ships oneTBB, which dropped the tbb::internal namespace. Both are installed under the same library name, so once one copy is in the process the loader binds anything that asks for that name to it:

platform	CmdStan 2.40.0	RcppParallel 6.2.1 binary
macOS	libtbb.dylib (install name @rpath/libtbb.dylib)	libtbb.dylib (same install name)
Windows	tbb.dll	lib/x64/tbb.dll
Linux	libtbb.so.2	lib/libtbb.so.2

RcppParallel renames its oneTBB to libtbb.so.2 on Linux on purpose, for packages built against older versions. Windows resolves a DLL already in the process by module name, so it collides the way macOS does. Linux matches by path and SONAME, so both copies may load there; untested.

Sampling is unaffected: the executable runs in a child process. Only the model methods (log_prob(), grad_log_prob(), unconstrain_variables(), ...) and $expose_functions() are exposed.

What to do

Build the R-loaded code against RcppParallel's oneTBB whenever RcppParallel is installed. Stan Math supports oneTBB through TBB_INTERFACE_NEW, and CmdStan's makefiles take an external TBB through TBB_INC and TBB_LIB, so rcpp_source_stan() can pass those three make variables when it asks make for the compile and link flags and leave everything else alone. Verified on macOS: with RcppParallel's include and lib directories substituted, the methods build in 18 s and load in both orders, and log_prob() and grad_log_prob() match a clean session. $expose_functions() goes through the same function.

The real fix is still upstream: CmdStan bundling oneTBB, the generation everything else now ships. Until then this makes RcppParallel's TBB the one the R process uses whenever RcppParallel is around, and CmdStan's when it isn't.

Reproduce

loadNamespace("RcppParallel")
library(cmdstanr)
mod <- cmdstan_model(write_stan_file("
data { int<lower=0> N; array[N] int<lower=0,upper=1> y; }
parameters { real<lower=0,upper=1> theta; }
model { theta ~ beta(1,1); y ~ bernoulli(theta); }
"))
fit <- mod$sample(data = list(N = 10, y = c(0,1,0,0,0,0,0,0,0,1)), chains = 1)
fit$init_model_methods()

[jgabry]

Other possible directions

Stan Math supports the newer TBB API through TBB_INTERFACE_NEW and can build against an external TBB using TBB_INC and TBB_LIB, although CmdStan currently defaults to its bundled TBB 2020.3.

One option may be to compile only the R-loadable libraries against a oneTBB runtime compatible with RcppParallel. RcppParallel exposes supported compilation and linking helpers, so CmdStanR may be able to use those rather than depending on private installation paths. This would likely make RcppParallel a dependency of compiled model methods and expose_functions(), and it would need testing across platforms and RcppParallel configurations.

Other possible solutions may involve changing the runtime library identity or waiting until CmdStan's default TBB changes. Input from Stan Math and RcppParallel maintainers would be useful before choosing an approach.