Determine project goals

[tarcieri]

At a high level, the goal of this project is to make it easy to develop software in Rust which, when compiled, can be independently verified to determine that a given binary was produced from a particular source code input.

To accomplish this, compiling Rust code must produce byte-for-byte identical results given a particular source code input, allowing multiple third parties to come to a consensus on a “correct” result.

How exactly this should be accomplished is debatable, and this issue is a place to discuss potential designs.

See the previous Secure Code WG issue on this topic for more background:

rust-secure-code/wg#28

Prior Art

• reprotest (Python): reproducibility testing tool
• diffoscope (Python): binary diffing tool

[dongcarl]

This is super important for rust binary security and I'm overjoyed that someone's working on this!

I believe that once this is finished, release engineers can combine this with Guix's Rust bootstrap to have bootstrappable+reproducible builds, effectively mitigating "Trusting Trust".

@tarcieri I'm not 100% sure what your game plan is yet, but there might be a few good ideas we can borrow from Guix for determinism. One of those is building in Linux containers (cgroups+namespaces) where only the build time dependencies are available. This ensures that no unintended sources of non-determinism leaks in. Just realized that this probably needs to support building on non-Linux 😅

[tarcieri]

@dongcarl some what you mentioned calls to mind some important goals this tool should have (IMO):

• Portability: ideally this tool should work anywhere Cargo itself works
• Simplicity: there are a lot of ways to eliminate nondeterminism in builds, including running builds inside specially configured VMs or containers. However, given upstream work in rustc on reproducible builds (e.g. Bit-for-bit deterministic / reproducible builds rust-lang/rust#34902) such extreme measures should not be necessary
• Composability: as much as possible, cargo repro should strive to interoperate well with (and avoid duplicating efforts of) other popular Rust build tooling, e.g. interoperating with cross to support reproducible cross-compilation. This suggests a lighthanded approach is probably best, i.e. other tools like cross which manage build containers should drive cargo repro, rather than cargo repro managing containers/VMs itself.

[dongcarl]

@tarcieri Glad I somewhat helped 😄. However, I feel like I'm missing a bit of context here. Esp. w/re interactions with reproducible builds in rustc. Am I correct in understanding that this project will be building on top of the reproducible builds effort in rustc, and the only sources of non-reproducibility that will be handled in this project will be those introduced by cargo? Please let me know where I'm misunderstanding! ☺️

[tarcieri]

@dongcarl that's the basic idea, yes. The idea is to provide an easy-to-use, cargo-driven workflow which leverages the ongoing work in rustc to create builds which are fully deterministic, and ideally a simple method for testing that a build is reproducible.

[dongcarl]

Hmmm... Perhaps not relevant, but off the top of my head w/re testing for reproducibility, one thing that needs to be in the design is how we'll represent the "inputs" (perhaps source code + deps + build-tool versions + build configuration) to the build and its outputs such that they're easily comparable without bit-for-bit diffing.

In this context, I think this project''s overall invariant to be maintained is that the same inputs produce the same outputs. I believe that sha256 would work well to represent both the inputs and the outputs. (This is what NixOS and Gitian does I believe)

What would need to be spec'd out is the exact structure of the preimage or what exactly constitutes the "input", as in we need to decide what the input hash "commits" to and the exact ordering.

[tarcieri]

one thing that needs to be in the design is how we'll represent the "inputs" (perhaps source code + deps + build-tool versions + build configuration) to the build and its outputs such that they're easily comparable without bit-for-bit diffing

Excellent point! Somewhat related to this is @Shnatsel's rust-audit tool, which is capable of embedding/extracting (via the unpublished auditable crate) Cargo.lock files in Rust binaries, which at least addresses the particular dependency set at the time a binary was built (and also includes nice details including digests of the dependencies)

Perhaps this is a question for @Shnatsel, but it seems to me like it'd be nice if his tool could be extended to include all parameters needed to reproduce a build, e.g. by collecting a manifest of those parameters and serializing them as another TOML file to also include in the resulting binary.

Another important question is: what's an exhaustive list of those parameters, and which ones can presently be supplied/overridden when cargo invokes rustc? This post contains a spitball list of such parameters, but though they test them out empirically, their process was much more black box and used QEMU to drive the build, so it's unclear which of the ones checked/permuted are actually relevant to rustc.

[kpcyrd]

I've worked on reproducible rust as part of the reproducible builds project, from my point of view there's mostly one issue left that would need some love (besides occasional regressions and issues in crates): rust-lang/cargo#5505

With that patch in place we're always going to get the same binary with a plain cargo build, given:

• the rustc binary is identical
• cargo build --locked is used
• the build environment is sufficiently similar, for example if two systems have a different version of gcc installed you're most likely going to get a different result. In the same way, having a different openssl version installed is most likely also going vary the output

I think the 1st one could be addressed by interfacing with rustup, the 2nd one by making --locked the default (which I'm not sure why it isn't), but the 3rd is extremely system specific and there's probably no reasonable solution. For example debian and archlinux have their own system in place since they rebuild the build environment based on the information recorded in a buildinfo file.

I think auditable has a different scope, for reproducible builds we must have access to the source tar ball anyway which means we have a copy of the Cargo.lock that has been used.

I'm not sure a specific subcommand is needed since the 3rd step is most likely too much work and everything else can be done with vanilla cargo, but I would love to hear some specific usecases that I might be missing. :)

[tarcieri]

That's all great to hear, and given that it certainly sounds like much of the suggested project scope I gave earlier is unnecessary.

Given that it seems like the utility of a cargo-repro like tool might simply be nailing down all of the aforementioned details, i.e. provide simple automations for driving deterministic cargo builds (relying on cargo itself to do the actual builds), recording the relevant environmental/dependency details (ideally in the target binary), and a reprotest-like utility for verifying builds are reproducible and also giving helpful friendly errors for things like gcc or other system dependency mismatches (or perhaps simply providing a ready-made reprotest wrapper/recipe intended for reproducing Cargo projects).

Do any of those approaches sound interesting to you @kpcyrd? Does anything I just said overlap with existing efforts in Cargo proper?

Regarding this:

I think auditable has a different scope, for reproducible builds we must have access to the source tar ball anyway which means we have a copy of the Cargo.lock that has been used.

What I had in mind specifically was a tool intended to act on a Cargo project's source tree, be it in git, a tarball, or what have you. I suppose the simple solution to eliminating Cargo.lock discrepancies is the one you just mentioned: always check in the Cargo.lock of any project you intend to reproduce, and then it's a non-issue.

[Robbepop]

This has probably already been raised but I couldn't find it in this discussion.

Note that for example Rust's procedural macros could expand to non-deterministic code during compilation.
I noticed this while working on one that the parse tree I got from some dependency was created non-deterministically and thus my code that depended on its order was also generating non-deterministic output.
This could probably fixed to a certain extend by normalizing expanded Rust code (e.g. reordering certain things while retaining semantics etc.).

The compiler is certainly not going to be able to reason about the internal workings of procedural macros they are used pervasively within the Rust ecosystem and people won't stop using them.

Do you have any ideas about how you could solve this?

[tarcieri]

I can only offer the "Doctor, it hurts when I try to make reproducible builds of programs with nondeterministic procedural macros" solution of:

If you want your build to reproducible, all parts of it must be deterministic, including procedural macros.

I would suggest filing a bug against the dependency with nondeterministic codegen.

[tarcieri]

Based on the discussion above, I propose a cargo repro MVP do the following:

• cargo repro build collects environment details (e.g. OS, rustc version, C compiler version if cc crate is in Cargo.lock), records them in a file (e.g. Repro.toml), and then runs cargo build --locked
• cargo repro verify (or repro check? repro test?), when run in the parent directory of the source tree, and given a binary to reproduce as a command line option, along with a Repro.toml, will attempt to reproduce the binary. I think for an MVP the simplest way to get something out the door is to have this backed by reprotest, and collect some of the Rust reprotest recipes that have been floating around on various GitHub issues and forum posts.

Longer term it might be nice to get rid of the dependency on reprotest, but for me it seems like an OK way to start.

Does this approach sound good to everyone?

[dongcarl]

@tarcieri Sorry been quiet for a bit. I'm reading thru the comments and I want to understand what the diff between cargo repro and cargo+determinism patches would be? From my naive understanding, it seems that cargo repro's main automation would be managing non-Cargo.lock build-time dependencies, e.g. rustc version, is that correct?

[tarcieri]

@dongcarl that's what I'm proposing after discovering the current state of reproducibility of cargo builds. Still curious what people think about that.

If people feel that's worthwhile, it'd be good to enumerate specifically what's needed. I'm thinking things like:

• target/.rustc_info.json (or cross-compile targets)
• When cc present in Cargo.lock: C compiler used
• Directory where build was performed
• Additional environment details: OS, etc.

[gilescope]

If we had a tool that could pin point where we're introducing non-determinancy that would be great. If it could point the finger at an offending crate that would seem like a good start to me.