experiment: wasm execution engine

[AuHau]

This is just an experiment, not meant to be merged! Hence, only a draft PR.

tldr: I created a POC with a WASM runtime embedded in Bee. Modules execute locally. See Demo section at bottom for examples.

Motivation

During my Swarm days, I had this idea: what if there was a layer on top of Bee's DISC that would allow computation and interaction with DISC? Even if it would only be local execution on "your node", this would allow basically implementing all the higher-level protocols of Swarm like Manifests, Feeds, etc with it. Each protocol would have its own implementation module that would be uploaded to the network and, upon the user's request, downloaded and executed locally. This would add great flexibility in how these primitives would compose, iterate, and integrate.

I discussed this idea with @zelig and discovered that he has a similar idea. One overlap between our ideas was adding a safe execution engine. He originally mentioned using something like the V8 JavaScript engine, so I wanted to explore some areas around:

• integration - how it will incorporate into the existing codebase
• security - obviously embedding an engine in which untrusted code will be executed is a big security problem - how could this be handled?
• DevX - how will devs be using this?

This PR, and especially these notes, are what came out of this exploration.

My plan was simple. To include the execution engine in Bee and add an API call that lets you locally execute a content-addressed module uploaded in the Swarm network. This later expanded a bit into exploring how HTTP and browsers could integrate with the execution engine.

Engine

I started by deciding which engine to integrate. @zelig suggested originally V8 JavaScript engine. Quickly, I ran into the fact that to include that directly into Bee, its build setup would have to be tweaked to enable CGO_ENABLED, which is currently disabled. I assume that is not desired. Later, I will explain why it might not really be a blocker, though.

But even if we could include it, I am not sure JavaScript is the right tool for this task. It has a large footprint and might be unnecessarily complex to embed in Bee as an execution engine. I believe a tool that exists exactly for this purpose is WASM. While this technology is not that big on the web itself (it is mainly used as optimization modules for compute-heavy use cases like graphics, etc), it is used on the "backends," for example, "serverless" function execution, plugin systems, and similar. It was built to be fast, safe, compact, and portable, which fits our scope well. I decided to continue with WASM.

Integration and security

WASM is designed to be a sandboxed environment, but even with that goal, leaks and exploits can still happen, so I looked for an architectural solution to improve security with a defense-in-depth approach. The obviously naive approach to integrate the engine is to rely on the WASM sandbox itself for security, include it directly in the codebase, and run it inside the bee process. A more complex solution would be to use the same concept that Chrome uses for isolation of website's namespaces - running the engine in an isolated process outside of bee process. Since Bee manages its own wallet, this seems like a good idea.

For an out-of-process approach, there are at least two ways it could work.
The first option is that bee would include the engine code, and upon bee startup, it would spawn itself again with a special flag to initialize for "WASM-only support" and an IPC channel between the two processes.
The second option is to create a dedicated helper that wouldn't be limited by a language or other restrictions in the bee codebase (for example, CGO_ENABLED). This binary could then be shipped directly in the bee binary using Go's embed mechanism. On the one hand, this would require more complex handling, as the embedded binary would need to be extracted somewhere, but it would also offer greater flexibility and solve the two-binary packaging issue. We already have Bee's DATA_DIR, so we have a place where the WASM binary could live. Moreover, this engine binary probably won't be very big, so the total size of Bee's package shouldn't grow much.

There are also additional mechanisms that would improve the security hardening at the OS level for the out-of-process approach. They are platform-specific, though, which would introduce a maintenance burden, but to list a few:

• Linux: rlimits and seccomp-bpf
• Windows: AppContainer, Job Objects
• macOS: Seatbelt

When looking at the WASM engines, two options stood out:

• wazero - Golang native runtime. Seems like actively maintained although not a mainstream choice.
• wasmtime - Runtime written in Rust. Has a strong focus on security and supports fuel metering. Deployed at large production scale, for example, in Fastly or Shopify.

This leads to another consideration: capping execution time. Computation time is an obviously constrained resource that needs to be protected similarly to storage and bandwidth. Untrusted code can be malicious and could, for example, incorporate an infinite loop. A simple, naive approach is to add a watchdog that kills execution upon timeout. A more robust approach is fuel metering, which assigns a cost to each operation and tracks it over execution, with a budget set for each executed module. If the approach for execution in the engine should be changed from "local level" (approach of my POC - call API and compute something on your node) into "network level" (for example module that will dynamically route content in the network - hence a module that might be run in the whole network) - then fuel metering becomes a critical feature in order to not overload the network.

If this work should continue and aim for production release, I would suggest using the out-of-process approach with a Rust helper that uses the wasmtime engine. More thought would have to be put into designing an efficient IPC channel, as there might be intensive data transfers between these two points.

For my POC, for the sake of simplicity, I decided to use the wazero engine directly incorporated into the bee code.

This POC

This PR includes a POC that incorporates the Wazero WASM engine and an API route /@/<ref>, which allows execution of a WASM module that is uploaded under the address <ref>.

This alone doesn't allow anything interesting, as the module has no input or output. I therefore extended support for input data first to the HTTP request body, and later to include the request's method, body, and headers (only a limited subset). Then I added support for setting the body, headers, and status code for the HTTP response. This let me basically create an HTTP server inside the engine. This is demonstrated in one of the examples I created: webapp.

Later on, I explored how to expose Bee's internal functionality or potentially helpers inside the engine and created the following host module and functions that allow fetch/upload data in Swarm:

• bytesGet() / chunkGet() - for fetching data
• bytesPut() / chunkPut() - for uploading data
• executeModule() - invocation of another module allowing module compositions

There is another example related to encryption/content metadata dispatching which demonstrates the module composition using executeModule() that I created: dispatch.

For more details about the POC implementation there are some notes here:

• OpenAPI documentaiton of the /@/ endpoint
• README on the swarm module ABI

DevX

How will devs interact with such an execution engine? That is also something I set out to explore.
One advantage of using WASM is that many languages support it. So potential developers have rather big flexibility around what tools and what languages to use. On the other hand, as the execution environment has its own specifications around exposed host functions and helpers, it is actually beneficial to have dedicated SDKs that will provide bindings and potentially other helper functions.

I created one such SDK for AssemblyScript (TypeScript but for WASM) called bee-wasm-js. It incorporates a few helpful functionalities:

• Host functions and environment bindings
• AssemblyScript build command
• Local runtime harness used for testing without Bee node
• Utilities for uploading and execution of the module on Bee node

Demo

Example of a very simple module built with bee-wasm-js:

// assembly/index.ts
import { bytesGet, codeName, readInput, writeOutput, writeOutputString } from "bee-wasm-js/assembly";

export function _start(): void {
  const reference = readInput();            // the request body: a 32-byte reference
  const result = bytesGet(reference);       // fetch what it points at
  if (!result.ok) {
    writeOutputString("error: " + codeName(result.code));
    return;
  }
  writeOutput(result.unwrap());
}

If you want to run it, then make binary of this branch, and run a node at least in light mode.

To build it, upload it and execute it:

$ npm install bee-wasm-js
$ npx bee-wasm build assembly/index.ts -o build/module.wasm
$ REF=$(npx bee-wasm upload build/module.wasm)
$ npx bee-wasm exec "$REF" --input-hex <a reference to fetch>

I also built two more elaborate examples:

• dispatch - using the sub-execution of another module with the executeModule() call
• webapp - backend HTTP server

Resources

• https://github.com/AuHau/bee-wasm-js
• https://github.com/AuHau/bee-wasm-examples

Disclaimer

AI was used in this work. I reviewed part of the bee's initial groundwork, but then relaxed my oversight, as this is really an exploratory POC not meant to be merged in.