This post is best described as a technology demonstration; it melds together
web servers, plugins, WebAssembly, Go, Rust and ABIs. Here’s what it shows:

  • How to load WASM code with WASI in a Go environment and hook it up to a web
    server.
  • How to implement web server plugins in any language that can be compiled to
    WASM.
  • How to translate Go programs into WASM that uses WASI.
  • How to translate Rust programs into WASM that uses WASI.
  • How to write WAT (WebAssembly Text) code that uses WASI to interact with
    a non-JS environment.

We’re going to build a simple FAAS (Function as a Service) server in Go
that lets us write modules in any language that has a WASM
target. Comparing to existing technologies, it’s something
between GCP’s Cloud Functions, Cloud
Run
and good old CGI.

Design

Let’s start with a high-level diagram describing how the system works:

Diagram showing flow of events in this program, also described below

The steps numbered in the diagram are:

  1. The FAAS server receives an HTTP GET request, with a path consisting of
    a module name (func in the example in the diagram) and an arbitrary
    query string.
  2. The FAAS server finds and loads the WASM module corresponding to the module
    name it was provided, and invokes it with a description of the HTTP request.
  3. The module emits output to its stdout, which is captured by the FAAS server.
  4. The FAAS server uses the module’s stdout as the contents of an HTTP Response
    to the request it received.

The FAAS server

We’ll start our deep dive with the FAAS server itself (full code here). The
HTTP handling part is straightforward:

func httpHandler(w http.ResponseWriter, req *http.Request) {
  parts := strings.Split(strings.Trim(req.URL.Path, "/"), "/")
  if len(parts) < 1 {
    http.Error(w, "want /{modulename} prefix", http.StatusBadRequest)
    return
  }
  mod := parts[0]
  log.Printf("module %v requested with query %v", mod, req.URL.Query())

  env := map[string]string{
    "http_path":   req.URL.Path,
    "http_method": req.Method,
    "http_host":   req.Host,
    "http_query":  req.URL.Query().Encode(),
    "remote_addr": req.RemoteAddr,
  }

  modpath := fmt.Sprintf("target/%v.wasm", mod)
  log.Printf("loading module %v", modpath)
  out, err := invokeWasmModule(mod, modpath, env)
  if err != nil {
    log.Printf("error loading module %v", modpath)
    http.Error(w, "unable to find module "+modpath, http.StatusNotFound)
    return
  }

  // The module's stdout is written into the response.
  fmt.Fprint(w, out)
}

func main() {
  mux := http.NewServeMux()
  mux.HandleFunc("/", httpHandler)
  log.Fatal(http.ListenAndServe(":8080", mux))
}

This server listens on port 8080 (feel free to change this or make it
more configurable), and registers a catch-all handler for the root path. The
handler parses the actual request URL to find the module name. It then stores
some information to pass to the loaded module in the env map.

The loaded module is found with a filesystem lookup in the target directory
relative to the FAAS server binary. All of this is just for demonstration
purposes and can be easily changed, of course. The handler then calls
invokeWasmModule, which we’ll get to shortly. This function returns the
invoked module’s stdout, which the handler prints out into the HTTP response.

Running WASM code in Go

Given a WASM module, how do we run it programmatically in Go? There are several
high-quality WASM runtimes that work outside the browser environment, and many
of them have Go bindings; for example wasmtime-go. The one I like most,
however, is wazero; it’s a
zero-dependency, pure Go runtime that doesn’t have any prerequisites except
running a go get. Our FAAS server is using wazero to load and run
WASM modules.

Here’s invokeWasmModule:

// invokeWasmModule invokes the given WASM module (given as a file path),
// setting its env vars according to env. Returns the module's stdout.
func invokeWasmModule(modname string, wasmPath string, env map[string]string) (string, error) {
  ctx := context.Background()

  r := wazero.NewRuntime(ctx)
  defer r.Close(ctx)
  wasi_snapshot_preview1.MustInstantiate(ctx, r)

  // Instantiate the wasm runtime, setting up exported functions from the host
  // that the wasm module can use for logging purposes.
  _, err := r.NewHostModuleBuilder("env").
    NewFunctionBuilder().
    WithFunc(func(v uint32) {
      log.Printf("[%v]: %v", modname, v)
    }).
    Export("log_i32").
    NewFunctionBuilder().
    WithFunc(func(ctx context.Context, mod api.Module, ptr uint32, len uint32) {
      // Read the string from the module's exported memory.
      if bytes, ok := mod.Memory().Read(ptr, len); ok {
        log.Printf("[%v]: %v", modname, string(bytes))
      } else {
        log.Printf("[%v]: log_string: unable to read wasm memory", modname)
      }
    }).
    Export("log_string").
    Instantiate(ctx)
  if err != nil {
    return "", err
  }

  wasmObj, err := os.ReadFile(wasmPath)
  if err != nil {
    return "", err
  }

  // Set up stdout redirection and env vars for the module.
  var stdoutBuf bytes.Buffer
  config := wazero.NewModuleConfig().WithStdout(&stdoutBuf)

  for k, v := range env {
    config = config.WithEnv(k, v)
  }

  // Instantiate the module. This invokes the _start function by default.
  _, err = r.InstantiateWithConfig(ctx, wasmObj, config)
  if err != nil {
    return "", err
  }

  return stdoutBuf.String(), nil
}

Interesting things to note about this code:

  • wazero supports WASI, which has to be instantiated explicitly to be usable
    by the loaded modules.
  • A lot of the code deals with exporting logging functions from the host (the
    Go code of the FAAS server) to the WASM module.
  • We set up the loaded module’s stdout to be redirected to a buffer, and set up
    its environment variables to match the env map passed in.

There are several way for host code to interact with WASM modules using only the
WASI API and ABI. Here, we opt for using environment variables for input and
stdout for output, but there are other options (see the Other resources
section in the bottom for some pointers).

This is it – the whole FAAS server, about 100 LOC of commented Go code. Now
let’s move on to see some WASM modules this thing can load and run.

Writing modules in Go

We can compile Go code to WASM that uses WASI. Here’s a basic Go program that
emits a greeting and a listing of its environment variables to stdout:

package main

import (
  "fmt"
  "os"
)

func main() {
  fmt.Println("goenv environment:")

  for _, e := range os.Environ() {
    fmt.Println(" ", e)
  }
}

Until recently, the only way to compile Go code to WASM that works outside the
browser was by using the TinyGo compiler. In our
FAAS project structure, the invocation from the root directory is:

$ tinygo build -o target/goenv.wasm -target=wasi examples/goenv/goenv.go

Sharp-eyed readers will recall that the target/ directory is precisely where
the FAAS server looks for *.wasm files to load as modules. Now that we’ve
placed a module named goenv.wasm there, we’re ready to launch our server
with go run . in the root directory. We can issue a HTTP request to its
goenv module in a separate terminal:

$ curl "localhost:8080/goenv?foo=bar&id=1234"
goenv environment:
  http_method=GET
  http_host=localhost:8080
  http_query=foo=bar&id=1234
  remote_addr=127.0.0.1:59268
  http_path=/goenv

And looking at the terminal where the FAAS server runs we’ll see some logging
like:

2023/04/29 06:35:59 module goenv requested with query map[foo:[bar] id:[1234]]
2023/04/29 06:35:59 loading module target/goenv.wasm

As I’ve mentioned before, this was the main way to compile to WASI until
recently
. In the upcoming Go release (version 1.21), new support for the WASI
target is included in the main Go toolchain (the gc compiler) [1]. It’s
easy to try today either by building Go from source, or using gotip:

$ GOOS=wasip1 GOARCH=wasm gotip build -o target/goenv.wasm examples/goenv/goenv.go

(the wasip1 target name refers to “WASI Preview 1”)

Writing modules in Rust

Rust is another language that has good support for WASM and WASI in the build
system. After adding the wasm32-wasi target with rustup, it’s as simple
as passing the target name to cargo:

$ cargo build --target wasm32-wasi --release

The code is straightforward, similarly to the Go version:

use std::env;

fn main() {
    println!("rustenv environment:");

    for (key, value) in env::vars() {
        println!("  {key}: {value}");
    }
}

Writing modules in WebAssembly Text (WAT)

As we’ve seen, compiling Go and Rust code to WASM is fairly easy; looking for a
challenge, let’s write a module in WAT! As I’ve written before, I enjoy
writing directly in WAT; it’s educational, and produces remarkably compact
binaries.

The “educational” aspect quickly becomes apparent when thinking about our task.
How exactly am I supposed to write to stdout or read environment variables using
WASM? This is where WASI comes in. WASI defines both an API and ABI, both of
which will be visible in our sample. The following shows some code snippets with
explanations; for the full code check out the sample repository.

First, I want to show how output to stdout is done; we start by importing
the fd_write WASI system call:

(import "wasi_snapshot_preview1" "fd_write" (func $fd_write (param i32 i32 i32 i32) (result i32)))

Apparently, it has four i32 parameters and returns an i32; what do all
of these mean? Unfortunately, WASI documentation could use a lot of work; the
resources I found useful are [2]:

  1. Legacy preview 1 specs
  2. C header descriptions of these functions

With this in hand, I was able to concoct a useful println equivalent in
WAT that uses fd_write under the hood:

;; println prints a string to stdout using WASI.
;; It takes the string's address and length as parameters.
(func $println (param $strptr i32) (param $len i32)
    ;; Print the string pointed to by $strptr first.
    ;;   fd=1
    ;;   data vector with the pointer and length
    (i32.store (global.get $datavec_addr) (local.get $strptr))
    (i32.store (global.get $datavec_len) (local.get $len))
    (call $fd_write
        (i32.const 1)
        (global.get $datavec_addr)
        (i32.const 1)
        (global.get $fdwrite_ret)
    )
    drop

    ;; Print out a newline.
    (i32.store (global.get $datavec_addr) (i32.const 850))
    (i32.store (global.get $datavec_len) (i32.const 1))
    (call $fd_write
        (i32.const 1)
        (global.get $datavec_addr)
        (i32.const 1)
        (global.get $fdwrite_ret)
    )
    drop
)

This uses some globals that you’ll have to look up in the full code sample
if you’re interested. Here’s another helper function that prints out a
zero-terminated string to stdout:

;; show_env emits a single env var pair to stdout. envptr points to it,
;; and it's 0-terminated.
(func $show_env (param $envptr i32)
    (local $i i32)
    (local.set $i (i32.const 0))

    ;; for i = 0; envptr[i] != 0; i++
    (loop $count_loop (block $break_count_loop
        (i32.eqz (i32.load8_u (i32.add (local.get $envptr) (local.get $i))))
        br_if $break_count_loop

        (local.set $i (i32.add (local.get $i) (i32.const 1)))
        br $count_loop
    ))

    (call $println (local.get $envptr) (local.get $i))
)

The fun part about writing assembly is that there are no abstractions.
Everything is out in the open. You know how strings are typically represented
using either zero termination (like in C) or a (start, len) pair?
In manual WAT code that uses WASI we have the pleasure of using both approaches
in the same program :-)

Finally, our main function:

(func $main (export "_start")
    (local $i i32)
    (local $num_of_envs i32)
    (local $next_env_ptr i32)

    (call $log_string (i32.const 750) (i32.const 19))

    ;; Find out the number of env vars.
    (call $environ_sizes_get (global.get $env_count) (global.get $env_len))
    drop

    ;; Get the env vars themselves into memory.
    (call $environ_get (global.get $env_ptrs) (global.get $env_buf))
    drop

    ;; Print out the preamble
    (call $println (i32.const 800) (i32.const 19))

    ;; for i = 0; i != *env_count; i++
    ;;   show env var i
    (local.set $num_of_envs (i32.load (global.get $env_count)))
    (local.set $i (i32.const 0))
    (loop $envvar_loop (block $break_envvar_loop
        (i32.eq (local.get $i) (local.get $num_of_envs))
        (br_if $break_envvar_loop)

        ;; next_env_ptr <- env_ptrs[i*4]
        (local.set
            $next_env_ptr
            (i32.load (i32.add  (global.get $env_ptrs)
                                (i32.mul (local.get $i) (i32.const 4)))))

        ;; print out this env var
        (call $show_env (local.get $next_env_ptr))

        (local.set $i (i32.add (local.get $i) (i32.const 1)))
        (br $envvar_loop)
    ))
)

We can now compile this WAT code into a FAAS module and re-run the server:

$ wat2wasm examples/watenv.wat -o target/watenv.wasm
$ go run .

Let’s try it:

$ curl "localhost:8080/watenv?foo=bar&id=1234"
watenv environment:
http_host=localhost:8080
http_query=foo=bar&id=1234
remote_addr=127.0.0.1:43868
http_path=/watenv
http_method=GET

WASI: API and ABI

I’ve mentioned the WASI API and ABI earlier; now it’s a good time to explain
what that means. An API is a set of functions that programs using WASI have
access to; one can think of it as a standard library of sorts. Go programmers
have access to the fmt package and the Println function within it.
Programs targeting WASI have access to the fd_write system call in the
wasi_snapshow_preview1 module, and so on. The API of fd_write also
defines how this function takes parameters and what it returns. Our sample uses
three WASI functions: fd_write, environ_sizes_get and environ_get.

An ABI is a little bit less familiar to most programmers; it’s the run-time
contract between a program and its environment. The WASI ABI is currently
unstable and is described here. In
our program, the ABI manifests in two ways:

  1. The main entry point we export is the _start function. This is
    automatically called by a WASI-supporting host after setup.
  2. Our WASM code exports its linear memory to the host with
    (memory (export "memory") 1). Since WASI APIs require passing pointers
    to memory, both the host and the WASM module need a shared understanding
    of how to access this memory.

Naturally, both the Go and Rust implementations of FAAS modules comply to the
WASI API and ABI, but this is hidden by the compiler from programmers. In the
Go program, for example, all we need to do is write a main function as usual
and therein emit to stdout using Println. The Go compiler will properly
export _start and memory:

$ wasm-objdump -x target/goenv.wasm

... snip

Export[2]:
 - func[1028] <_rt0_wasm_wasip1> -> "_start"
 - memory[0] -> "memory"

... snip

And will properly hook things up to call our code from _start, etc.

WASI and plugins

The FAAS server presented in this post is clearly an example of developing
plugins using WASM and WASI. This is an emerging and exciting area in
programming and lots of progress is being made on multiple fronts. Right now,
WASI modules are limited to interacting with the environment via means like
environment variables and stdin/stdout; while this is fine for interacting
with the outside world, for host-to-module communication it’s not amazing, in
my experience. Therefore the WASM standards committee is working of further
improvements to WASI that may include sockets and other means of passing data
between hosts and modules.

In the meantime, projects are making do with what they have. For example, the
sqlc Go package supports WASM plugins. The communication
with plugins happens as follows: the host encodes a command into a protobuf
and emits it to the plugin’s stdin; it then reads the plugin’s stdout for a
protobuf-encoded response.

Other projects are taking more maverick approaches; for example, the Envoy
proxy
supports WASM plugins by defining a custom
API and ABI between the host and WASM modules. I’ll probably write more about
this in a later post.

Other resources

Here are some additional resources on the same topic as this post:


Read More