From 5ac9ffb0bfcbb7fbbdc8042b9a948d7c0592adbb Mon Sep 17 00:00:00 2001 From: Jesse Stimpson Date: Fri, 6 Mar 2026 08:46:37 -0500 Subject: [PATCH 1/4] Improve compile time behavior to support multiversion deployments --- .github/workflows/ci.yml | 2 +- README.md | 8 +- notes/configuration.md | 155 +++++++++++++++++++++++++ rebar.config | 1 + rebar.config.script | 90 ++++++++------ src/erlfdb.app.src | 2 +- src/erlfdb.erl | 2 +- src/erlfdb_nif.erl | 30 +++-- test/erlfdb_02_anon_fdbserver_test.erl | 14 +-- 9 files changed, 244 insertions(+), 60 deletions(-) create mode 100644 notes/configuration.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8c99124..fde6a43 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -39,7 +39,7 @@ jobs: otp-version: ${{matrix.otp}} rebar3-version: "3.24.0" - name: Run rebar3 fmt check for Erlang files - run: ERLFDB_ASSERT_FDBCLI=0 rebar3 fmt --check + run: ERLFDB_ASSERT_API_VERSION=0 rebar3 fmt --check if: ${{ matrix.fmt }} - name: Run clang-format check for C files uses: jidicula/clang-format-action@v4.11.0 diff --git a/README.md b/README.md index 3d682c6..fe40c0a 100644 --- a/README.md +++ b/README.md @@ -109,12 +109,12 @@ against the most recent supported version of FoundationDB. ### Bypassing dependency checks -When you execute a rebar command, erlfdb attempts to detect the version of the fdbcli -installed on your system. If it cannot be detected, the rebar command is aborted. +When you execute a rebar command, erlfdb attempts to detect the FDB API version +via `fdbcli`. If it cannot be determined, the rebar command is aborted. -To disable the abort, use `ERLFDB_ASSERT_FDBCLI=0`. For example, you can safely use +To disable the abort, use `ERLFDB_ASSERT_API_VERSION=0`. For example, you can safely use this for the `fmt` command, which does not do a compile action. ```bash -ERLFDB_ASSERT_FDBCLI=0 rebar3 fmt +ERLFDB_ASSERT_API_VERSION=0 rebar3 fmt ``` diff --git a/notes/configuration.md b/notes/configuration.md new file mode 100644 index 0000000..216179f --- /dev/null +++ b/notes/configuration.md @@ -0,0 +1,155 @@ +# Configuration + +erlfdb supports compile-time and runtime configuration. Compile-time options are resolved in `rebar.config.script` during the build. Runtime options are set via application environment variables, typically in a `sys.config` file. + +## Compile-Time Configuration + +The following OS environment variables are read by `rebar.config.script` via `os:getenv/1` during compilation. These must be environment variables (rather than Erlang compile-time defines) because the script runs before Erlang compilation begins — it is responsible for producing the `erl_opts` and `port_env` that drive the build. The purpose of these options is to provide the developer with control over which libfdb_c function calls exist in the NIF library's symbol table, so that the intended library can be loaded at runtime. + +### `ERLFDB_INCLUDE_DIR` + +Path to the directory containing the FoundationDB C API header files. Defaults to `/usr/local/include`. + +```shell +export ERLFDB_INCLUDE_DIR=/opt/foundationdb/include +``` + +### `ERLFDB_COMPILE_API_VERSION` + +When set, the build uses this value directly as the compile-time FDB API version, bypassing `fdbcli` checks. This is useful in environments where `fdbcli` is not installed on the build host but the FoundationDB client library and headers are available (e.g. multi-version support, cross-compilation or CI). erlfdb will compile only with features that are supported in this version, and will throw a `badarg` error when calling code executes an erlfdb function that is not supported. For example, calling `erlfdb:get_mapped_range/4` when compiled with version 710. + +```shell +export ERLFDB_COMPILE_API_VERSION=730 +``` + +### `ERLFDB_FDBCLI` + +Path to the `fdbcli` executable. If not set, erlfdb searches `PATH` and `/usr/local/bin`. The build script uses `fdbcli --version` to detect the FDB protocol version, which determines the C API version used during compilation. Ignored when `ERLFDB_COMPILE_API_VERSION` is set. + +```shell +export ERLFDB_FDBCLI=/opt/foundationdb/bin/fdbcli +``` + +### `ERLFDB_ASSERT_API_VERSION` + +When set to `"0"`, the build will continue with a warning if the FDB API version cannot be determined, instead of aborting. By default (any other value, or unset), an undetectable API version is a fatal error. + +```shell +# Allow the build to proceed without a detected API version +export ERLFDB_ASSERT_API_VERSION=0 +``` + +## Runtime Configuration + +Runtime configuration is set via the `erlfdb` application environment, typically in a `sys.config` file or equivalent. + +### `api_version` + +The FoundationDB C API version to use at runtime. Defaults to the value of the `erlfdb_compile_time_api_version` compile-time macro, which is auto-detected from `fdbcli` during the build, or from `ERLFDB_COMPILE_API_VERSION`. + +```erlang +[ + {erlfdb, [ + {api_version, 730} + ]} +]. +``` + +### `network_options` + +A proplist of options passed to the FoundationDB client network layer during NIF initialization. Your options are merged with erlfdb's defaults, with your values taking precedence. Setting any option to `false` causes it to be removed from the resolved list, making it a no-op when options are applied to FDB. This can be used to disable a default. + +The resolved options are stored in the `network_options_resolved` application env var and can be inspected at runtime: + +```erlang +application:get_env(erlfdb, network_options_resolved). +``` + +#### Defaults + +```erlang +[ + {callbacks_on_external_threads, true}, + {external_client_library, {erlfdb_network_options, compile_time_external_client_library, []}}, + {client_threads_per_version, 1} +]. +``` + +#### Key Options + +##### `callbacks_on_external_threads` + +When `true`, allows libfdb_c to execute future callbacks on external threads. This increases throughput of future resolution and is recommended for erlfdb. Default: `true`. + +##### `external_client_library` + +Path (as a binary string) to the `libfdb_c` dynamic library. libfdb_c copies this library N times to temporary files so they can be individually loaded by external client threads. + +The value may also be an `{M, F, A}` tuple. The function is called when the NIF is loaded and must return a binary string. Default: `{erlfdb_network_options, compile_time_external_client_library, []}`, which returns the library path detected at compile time. + +##### `client_threads_per_version` + +The number of client threads to create per dynamic library. This is the primary knob for scaling FDB client throughput horizontally. + +- When set to `1`, only the local client thread is used. +- When set to `N > 1`, N external client threads are created in addition to the main thread. + +The value may also be an `{M, F, A}` tuple. The function is called when the NIF is loaded and must return a positive integer. + +> #### Note {: .info} +> For each thread, libfdb_c creates a TCP connection for each coordinator in the cluster file. Choose a value sufficient for your workload but no larger. + +Default: `1`. + +Example scaling to 2 threads: + +```erlang +[ + {erlfdb, [ + {network_options, [ + {client_threads_per_version, 2} + ]} + ]} +]. +``` + +##### `external_client_directory` + +Path (as a binary string) to a directory containing additional `libfdb_c` dynamic libraries for multi-version client support. When set, libfdb_c loads all client libraries found in this directory, enabling connections to clusters running different FoundationDB versions. Each library version is loaded alongside the primary client. + +```erlang +[ + {erlfdb, [ + {network_options, [ + {external_client_directory, <<"/usr/lib/foundationdb/multiversion">>} + ]} + ]} +]. +``` + +##### `trace_enable` + +A binary string path to a directory where client trace files will be written. The directory must exist and be writable. + +##### `trace_format` + +The format of trace files. Supported values: `<<"xml">>` (default) and `<<"json">>`. + +```erlang +[ + {erlfdb, [ + {network_options, [ + {trace_enable, <<"/var/log/fdb_traces">>}, + {trace_format, <<"json">>} + ]} + ]} +]. +``` + +#### All Supported Network Options + +The full set of network options corresponds to the [FoundationDB C API network options](https://apple.github.io/foundationdb/api-c.html#c.fdb_network_set_option). + +Any option value may be set to `false` to disable it, even if it was set in the defaults. When `false`, the option key and value do not appear in the resolved options. + +Any option value may be set to `{M, F, A}` tuple to acquire the value at runtime. The option key and value will appear in the resolved options. diff --git a/rebar.config b/rebar.config index 7f339e2..01e6c06 100644 --- a/rebar.config +++ b/rebar.config @@ -24,6 +24,7 @@ <<"README.md">>, <<"LICENSE">>, <<"CHANGELOG.md">>, + <<"notes/configuration.md">>, <<"notes/thread-design.md">>, <<"notebooks/kv_queue.livemd">>, <<"notebooks/tutorial-elixir.livemd">> diff --git a/rebar.config.script b/rebar.config.script index 582e201..4c76ab9 100644 --- a/rebar.config.script +++ b/rebar.config.script @@ -1,21 +1,12 @@ FDBReleasesBaseUrl = "https://api.github.com/repos/apple/foundationdb/releases". -IncludeDir = "/usr/local/include". +IncludeDir = case os:getenv("ERLFDB_INCLUDE_DIR") of + false -> "/usr/local/include"; + Dir -> Dir +end. % Detect rebar3 or mix RebarApi = {error, nofile} =/= code:ensure_loaded(rebar_api). -% Look for fdbcli on PATH, but also check /usr/local/bin explicitly since that is the common location -FdbCli = - case os:find_executable("fdbcli") of - false -> - case os:find_executable("fdbcli", "/usr/local/bin") of - false -> "fdbcli"; - Exec -> Exec - end; - Exec -> - Exec - end. - % Logs to stdout/stderr from either rebar3 or mix Log = fun(Level, Fmt, Args) -> if @@ -38,24 +29,48 @@ Log = fun(Level, Fmt, Args) -> end end. +% Look for fdbcli on PATH, but also check /usr/local/bin explicitly since that is the common location. +% Override with ERLFDB_FDBCLI env var to point at a specific version's fdbcli. % Check architecture against regex IsArch = fun(ArchRe) -> match =:= re:run(erlang:system_info(system_architecture), ArchRe, [{capture, none}]) end. -% Hacky means to extract API version from fdbcli protocol version output -% See https://github.com/apple/foundationdb/blob/master/flow/ProtocolVersion.h -MaxAPIVersion = - begin - VsnInfo = os:cmd(FdbCli ++ " --version"), - case re:run(VsnInfo, "protocol ([a-f0-9]*)", [{capture, [1], list}]) of - {match, [ProtocolStr]} -> - ProtocolVsn = list_to_integer(ProtocolStr, 16), - APIVersionBytes = (ProtocolVsn band 16#0000000FFF00000) bsr 20, - list_to_integer(integer_to_list(APIVersionBytes, 16)); - nomatch -> - undefined - end +CompileAPIVersion = + case os:getenv("ERLFDB_COMPILE_API_VERSION") of + false -> + % Look for fdbcli on PATH, but also check /usr/local/bin explicitly since that is the common location. + % Override with ERLFDB_FDBCLI env var to point at a specific version's fdbcli. + FdbCli = + case os:getenv("ERLFDB_FDBCLI") of + false -> + case os:find_executable("fdbcli") of + false -> + case os:find_executable("fdbcli", "/usr/local/bin") of + false -> "fdbcli"; + Exec -> Exec + end; + Exec -> + Exec + end; + Path -> + Path + end, + + % Hacky means to extract API version from fdbcli protocol version output + % See https://github.com/apple/foundationdb/blob/master/flow/ProtocolVersion.h + VsnInfo = os:cmd(FdbCli ++ " --version"), + case re:run(VsnInfo, "protocol ([a-f0-9]*)", [{capture, [1], list}]) of + {match, [ProtocolStr]} -> + ProtocolVsn = list_to_integer(ProtocolStr, 16), + APIVersionBytes = (ProtocolVsn band 16#0000000FFF00000) bsr 20, + list_to_integer(integer_to_list(APIVersionBytes, 16)); + nomatch -> + undefined + end; + ApiVsnStr -> + Log(info, "Using ERLFDB_COMPILE_API_VERSION=~s", [ApiVsnStr]), + list_to_integer(ApiVsnStr) end. % Make an HTTP GET request, expect JSON body @@ -181,8 +196,8 @@ LogAssetsMap = fun ) end. -% If MaxAPIVersion undetected, print helpful informatiom about where to download the latest FDB Release -case MaxAPIVersion of +% If CompileAPIVersion undetected, print helpful informatiom about where to download the latest FDB Release +case CompileAPIVersion of undefined -> Log(info, "Checking for latest FoundationDB release...", []), {ok, _} = application:ensure_all_started(inets), @@ -200,14 +215,15 @@ case MaxAPIVersion of "The foundationdb-clients package is required to compile erlfdb. Please visit~n~n https://github.com/apple/foundationdb/releases~n" end, [Log(info, Message, []) || Message =/= undefined], - AssertFdbCli = "0" =/= os:getenv("ERLFDB_ASSERT_FDBCLI"), + AssertApiVersion = "0" =/= os:getenv("ERLFDB_ASSERT_API_VERSION"), if - AssertFdbCli -> - Log(abort, "Error: fdbcli not found on PATH.", []); + AssertApiVersion -> + Log(abort, "Error: FDB API version could not be determined. Set ERLFDB_COMPILE_API_VERSION or install fdbcli.", []); true -> - Log(error, "fdbcli not found on PATH. Continuing anyway...", []) + Log(error, "FDB API version could not be determined. Continuing anyway...", []) end; _ -> + Log(info, "Compiling with FDB_API_VERSION=~p", [CompileAPIVersion]), ok end. @@ -224,7 +240,7 @@ DynamicLibraryDir = filename:dirname(DynamicLibrary). [ {erl_opts, ErlOpts ++ [ - {d, erlfdb_api_version, MaxAPIVersion}, + {d, erlfdb_compile_time_api_version, CompileAPIVersion}, {d, erlfdb_compile_time_external_client_library, DynamicLibrary} ]}, {port_env, [ @@ -233,8 +249,8 @@ DynamicLibraryDir = filename:dirname(DynamicLibrary). "CFLAGS", "$CFLAGS -I"++IncludeDir++" -Ic_src/ -g -Wall -Werror " ++ (if - MaxAPIVersion < 730 -> - "-DFDB_API_VERSION=" ++ integer_to_list(MaxAPIVersion); + is_integer(CompileAPIVersion) -> + "-DFDB_API_VERSION=" ++ integer_to_list(CompileAPIVersion); true -> "-DFDB_USE_LATEST_API_VERSION=1" end) @@ -242,12 +258,12 @@ DynamicLibraryDir = filename:dirname(DynamicLibrary). { "(linux|solaris|freebsd|netbsd|openbsd|dragonfly|gnu)", "LDFLAGS", - "$LDFLAGS -L"++DynamicLibraryDir++" -lfdb_c" + "$LDFLAGS -Wl,-rpath,\\$ORIGIN -Wl,-rpath,"++DynamicLibraryDir++" -L"++DynamicLibraryDir++" -lfdb_c" }, { "(darwin)", "LDFLAGS", - "$LDFLAGS -rpath "++DynamicLibraryDir++" -L"++DynamicLibraryDir++" -lfdb_c" + "$LDFLAGS -rpath @loader_path -rpath "++DynamicLibraryDir++" -L"++DynamicLibraryDir++" -lfdb_c" } ]} ] ++ CONFIG1. diff --git a/src/erlfdb.app.src b/src/erlfdb.app.src index fb85d16..9fe0683 100644 --- a/src/erlfdb.app.src +++ b/src/erlfdb.app.src @@ -12,7 +12,7 @@ {application, erlfdb, [ {description, "Erlang client for FoundationDB"}, - {vsn, "1.0.0"}, + {vsn, "1.1.0"}, {registered, []}, {applications, [kernel, stdlib]}, {maintainers, ["Jesse Stimpson"]}, diff --git a/src/erlfdb.erl b/src/erlfdb.erl index 7a7cd15..953217f 100644 --- a/src/erlfdb.erl +++ b/src/erlfdb.erl @@ -1194,7 +1194,7 @@ get_range_startswith(DbOrTx, Prefix, Options) -> -if(?DOCATTRS). -doc """ -Equivalent to `get_mapped_range(Tx, StartKey, EndKey, []).` +Equivalent to `get_mapped_range(Tx, StartKey, EndKey, Mapper, []).` This function never returns a `t:fold_future/0`. Use `[{wait, false}]` with `get_mapped_range/5` if you want future semantics. diff --git a/src/erlfdb_nif.erl b/src/erlfdb_nif.erl index 4490a2b..41ce8e0 100644 --- a/src/erlfdb_nif.erl +++ b/src/erlfdb_nif.erl @@ -30,6 +30,7 @@ The NIF wrapper around all FoundationDB C API function calls. -export([ ohai/0, + get_default_api_version/0, get_max_api_version/0, future_cancel/1, @@ -80,11 +81,7 @@ The NIF wrapper around all FoundationDB C API function calls. error_predicate/2 ]). --ifdef(erlfdb_api_version). --define(DEFAULT_API_VERSION, ?erlfdb_api_version). --else. --define(DEFAULT_API_VERSION, 730). --endif. +-define(DEFAULT_API_VERSION, ?erlfdb_compile_time_api_version). -export_type([ atomic_mode/0, @@ -237,7 +234,10 @@ The NIF wrapper around all FoundationDB C API function calls. ohai() -> foo. --spec get_max_api_version() -> {ok, integer()}. +-spec get_default_api_version() -> integer(). +get_default_api_version() -> ?DEFAULT_API_VERSION. + +-spec get_max_api_version() -> integer(). get_max_api_version() -> erlfdb_get_max_api_version(). @@ -594,6 +594,7 @@ init() -> NetOptionsDefaults = erlfdb_network_options:get_defaults(), + % Merge defaults Opts = case application:get_env(erlfdb, network_options) of {ok, O} when is_list(O) -> @@ -602,6 +603,7 @@ init() -> NetOptionsDefaults end, + % Follow MFAs Opts2 = lists:map( fun ({Name, {M, F, A}}) when @@ -615,20 +617,30 @@ init() -> Opts ), - application:set_env(erlfdb, network_options_resolved, Opts2), + % Remove options with `false` value. They are always a no-op + Opts3 = lists:filter( + fun + ({_Name, false}) -> false; + (_) -> true + end, + Opts2 + ), + + application:set_env(erlfdb, network_options_resolved, Opts3), + % Apply options to FDB lists:foreach( fun (Name) when is_atom(Name) -> ok = network_set_option(Name, <<>>); ({Name, false}) when is_atom(Name) -> - ok; + erlang:error(badarg); ({Name, true}) when is_atom(Name) -> ok = network_set_option(Name, <<>>); ({Name, Value}) when is_atom(Name) -> ok = network_set_option(Name, Value) end, - Opts2 + Opts3 ), ok = erlfdb_setup_network() diff --git a/test/erlfdb_02_anon_fdbserver_test.erl b/test/erlfdb_02_anon_fdbserver_test.erl index 5e96956..e435786 100644 --- a/test/erlfdb_02_anon_fdbserver_test.erl +++ b/test/erlfdb_02_anon_fdbserver_test.erl @@ -35,7 +35,7 @@ db_client_info_test() -> Db = erlfdb_sandbox:open(), Busyness = erlfdb:get_main_thread_busyness(Db), ?assert(is_float(Busyness)), - Vsn = erlfdb_nif:get_max_api_version(), + Vsn = erlfdb_nif:get_default_api_version(), if Vsn >= 730 -> Status = erlfdb:wait(erlfdb:get_client_status(Db)), @@ -92,7 +92,7 @@ get_range_test() -> ?assertEqual(KVs, GetRangeResult), - Vsn = erlfdb_nif:get_max_api_version(), + Vsn = erlfdb_nif:get_default_api_version(), if Vsn >= 730 -> @@ -123,7 +123,7 @@ interleaving_test() -> KVs = create_range(Tenant, <<"interleaving_test">>, N), Mapper = create_mapping_on_range(Tenant, <<"interleaving_test">>, N, <<"hello world">>), - Vsn = erlfdb_nif:get_max_api_version(), + Vsn = erlfdb_nif:get_default_api_version(), [R1, R2, R3, foobar, R4] = erlfdb:transactional(Tenant, fun(Tx) -> % F1 is a future doing a small get_range @@ -226,7 +226,7 @@ create_mapping_on_range(Tenant, Label, N, Message) -> % element selector syntax can be used. This test demonstrates the minimal set % of keys necessary to exercise the feature. get_mapped_range_minimal_test() -> - Vsn = erlfdb_nif:get_max_api_version(), + Vsn = erlfdb_nif:get_default_api_version(), if Vsn >= 730 -> @@ -252,7 +252,7 @@ get_mapped_range_minimal_test() -> end. get_mapped_range_continuation_test() -> - Vsn = erlfdb_nif:get_max_api_version(), + Vsn = erlfdb_nif:get_default_api_version(), if Vsn >= 730 -> N = 100, @@ -413,7 +413,7 @@ watch_to_test() -> receive {ResultRef, Result} -> ?assertMatch(<<"bar">>, Result) - after 1000 -> + after 2000 -> error(timeout) end. @@ -506,7 +506,7 @@ range_iterator_test() -> ), % GetMappedRange - Vsn = erlfdb_nif:get_max_api_version(), + Vsn = erlfdb_nif:get_default_api_version(), if Vsn >= 730 -> From bf0f47b0b11fe6e01cd7b0861f727bb944dba97b Mon Sep 17 00:00:00 2001 From: Jesse Stimpson Date: Fri, 6 Mar 2026 09:20:26 -0500 Subject: [PATCH 2/4] Add GH Action to exercise simple multiversion client configuration --- .github/workflows/ci.yml | 26 +++++++ test/multiversion/run_upgrade_test.sh | 108 ++++++++++++++++++++++++++ test/multiversion/verify_fdb.escript | 60 ++++++++++++++ 3 files changed, 194 insertions(+) create mode 100755 test/multiversion/run_upgrade_test.sh create mode 100755 test/multiversion/verify_fdb.escript diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fde6a43..029f14f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -76,6 +76,32 @@ jobs: pip3 show foundationdb ./test/bindingtester/loop.sh if: ${{ matrix.bindingtester }} + multiversion: + runs-on: ubuntu-latest + strategy: + matrix: + include: + - otp: "27.0.1" + fdb_old: "7.2.2" + fdb_new: "7.3.62" + name: "linux / OTP ${{ matrix.otp }} / FDB ${{ matrix.fdb_old }} -> ${{ matrix.fdb_new }} (multiversion)" + env: + FDB_OLD: ${{ matrix.fdb_old }} + FDB_NEW: ${{ matrix.fdb_new }} + steps: + - uses: actions/checkout@v4 + - uses: erlef/setup-beam@v1 + with: + otp-version: ${{matrix.otp}} + rebar3-version: "3.24.0" + - name: Download FDB packages + run: | + wget https://github.com/apple/foundationdb/releases/download/${FDB_OLD}/foundationdb-clients_${FDB_OLD}-1_amd64.deb + wget https://github.com/apple/foundationdb/releases/download/${FDB_OLD}/foundationdb-server_${FDB_OLD}-1_amd64.deb + wget https://github.com/apple/foundationdb/releases/download/${FDB_NEW}/foundationdb-clients_${FDB_NEW}-1_amd64.deb + wget https://github.com/apple/foundationdb/releases/download/${FDB_NEW}/foundationdb-server_${FDB_NEW}-1_amd64.deb + - name: Run multiversion upgrade test + run: bash test/multiversion/run_upgrade_test.sh macos: runs-on: macos-latest strategy: diff --git a/test/multiversion/run_upgrade_test.sh b/test/multiversion/run_upgrade_test.sh new file mode 100755 index 0000000..2228da4 --- /dev/null +++ b/test/multiversion/run_upgrade_test.sh @@ -0,0 +1,108 @@ +#!/usr/bin/env bash +# +# Multi-version upgrade test for erlfdb. +# +# Validates that erlfdb compiled against an older FDB version can connect +# to both old and new servers via the FDB multi-version client mechanism. +# +# Environment variables (set by CI): +# FDB_OLD - old FDB version (e.g. "7.2.2") +# FDB_NEW - new FDB version (e.g. "7.3.62") +# +# Prerequisites: FDB .deb packages for both versions must be downloaded +# in the current directory before running this script. +# +set -euo pipefail + +: "${FDB_OLD:?FDB_OLD must be set (e.g. 7.2.2)}" +: "${FDB_NEW:?FDB_NEW must be set (e.g. 7.3.62)}" + +# Derive the compile-time API version from the old FDB major.minor. +# FDB 7.2.x -> API 720, 7.3.x -> API 730, etc. +FDB_OLD_MAJOR="${FDB_OLD%%.*}" +FDB_OLD_MINOR="${FDB_OLD#*.}"; FDB_OLD_MINOR="${FDB_OLD_MINOR%%.*}" +COMPILE_API_VERSION="${FDB_OLD_MAJOR}${FDB_OLD_MINOR}0" +PORT=4500 +TEST_DIR="/tmp/fdb-upgrade-test" +DATA_DIR="${TEST_DIR}/data" +LOG_DIR="${TEST_DIR}/logs" +CLUSTER_FILE="${TEST_DIR}/fdb.cluster" +MV_DIR="/opt/fdb-multiversion" +FDB_NEW_DIR="/opt/fdb-${FDB_NEW}" + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +PROJECT_DIR="$(cd "${SCRIPT_DIR}/../.." && pwd)" + +cleanup() { + echo "=== Cleanup ===" + kill "${FDB_PID:-}" 2>/dev/null || true + wait "${FDB_PID:-}" 2>/dev/null || true + rm -rf "${TEST_DIR}" +} +trap cleanup EXIT + +echo "=== Step 1: Install FDB ${FDB_OLD} (system version) ===" +sudo dpkg -i "foundationdb-clients_${FDB_OLD}-1_amd64.deb" +sudo dpkg -i "foundationdb-server_${FDB_OLD}-1_amd64.deb" +# Stop the auto-started fdbmonitor; we manage fdbserver directly +sudo systemctl stop foundationdb 2>/dev/null || sudo service foundationdb stop 2>/dev/null || true + +echo "=== Step 2: Extract FDB ${FDB_NEW} to ${FDB_NEW_DIR} ===" +sudo mkdir -p "${FDB_NEW_DIR}" +sudo dpkg-deb -x "foundationdb-clients_${FDB_NEW}-1_amd64.deb" "${FDB_NEW_DIR}" +sudo dpkg-deb -x "foundationdb-server_${FDB_NEW}-1_amd64.deb" "${FDB_NEW_DIR}" + +echo "=== Step 3: Set up multiversion client directory ===" +sudo mkdir -p "${MV_DIR}" +sudo cp /usr/lib/libfdb_c.so "${MV_DIR}/libfdb_c_${FDB_OLD}.so" +sudo cp "${FDB_NEW_DIR}/usr/lib/libfdb_c.so" "${MV_DIR}/libfdb_c_${FDB_NEW}.so" +ls -la "${MV_DIR}/" + +echo "=== Step 4: Compile erlfdb against FDB ${FDB_OLD} ===" +cd "${PROJECT_DIR}" +rm -f priv/erlfdb_nif.so c_src/*.o c_src/*.d +ERLFDB_INCLUDE_DIR=/usr/include \ +ERLFDB_COMPILE_API_VERSION="${COMPILE_API_VERSION}" \ + rebar3 compile + +echo "=== Step 5: Start fdbserver ${FDB_OLD} ===" +mkdir -p "${DATA_DIR}" "${LOG_DIR}" +echo "erlfdbmvtest:erlfdbmvtest@127.0.0.1:${PORT}" > "${CLUSTER_FILE}" + +/usr/sbin/fdbserver \ + -p "127.0.0.1:${PORT}" \ + -C "${CLUSTER_FILE}" \ + -d "${DATA_DIR}" \ + -L "${LOG_DIR}" & +FDB_PID=$! +echo "fdbserver ${FDB_OLD} started (PID ${FDB_PID})" +sleep 3 + +echo "=== Step 6: Initialize database ===" +/usr/bin/fdbcli -C "${CLUSTER_FILE}" --exec "configure new single ssd" +sleep 2 + +echo "=== Step 7: Pre-upgrade verification ===" +escript test/multiversion/verify_fdb.escript "${CLUSTER_FILE}" pre_upgrade "${MV_DIR}" "${COMPILE_API_VERSION}" + +echo "=== Step 8: Stop fdbserver ${FDB_OLD} ===" +kill "${FDB_PID}" +wait "${FDB_PID}" 2>/dev/null || true +unset FDB_PID +sleep 2 + +echo "=== Step 9: Start fdbserver ${FDB_NEW} ===" +"${FDB_NEW_DIR}/usr/sbin/fdbserver" \ + -p "127.0.0.1:${PORT}" \ + -C "${CLUSTER_FILE}" \ + -d "${DATA_DIR}" \ + -L "${LOG_DIR}" & +FDB_PID=$! +echo "fdbserver ${FDB_NEW} started (PID ${FDB_PID})" +sleep 5 + +echo "=== Step 10: Post-upgrade verification ===" +escript test/multiversion/verify_fdb.escript "${CLUSTER_FILE}" post_upgrade "${MV_DIR}" "${COMPILE_API_VERSION}" + +echo "" +echo "=== Multi-version upgrade test PASSED ===" diff --git a/test/multiversion/verify_fdb.escript b/test/multiversion/verify_fdb.escript new file mode 100755 index 0000000..95c9adb --- /dev/null +++ b/test/multiversion/verify_fdb.escript @@ -0,0 +1,60 @@ +#!/usr/bin/env escript +%%! -pa _build/default/lib/erlfdb/ebin + +-mode(compile). + +main([ClusterFile, Phase, MvDir, ApiVersionStr]) -> + ApiVersion = list_to_integer(ApiVersionStr), + io:format("[~s] Starting verification (api_version=~b)~n", [Phase, ApiVersion]), + + %% Load erlfdb app metadata (does NOT load any modules or the NIF) + ok = application:load(erlfdb), + + %% Configure multi-version client BEFORE any erlfdb module is loaded. + %% api_version must be compatible with both old and new FDB versions. + ok = application:set_env(erlfdb, api_version, ApiVersion), + ok = application:set_env(erlfdb, network_options, [ + {external_client_directory, list_to_binary(MvDir)}, + {external_client_library, false} + ]), + + %% Opening the database triggers NIF loading -> init/0 reads our app env, + %% configures the multi-version client, and starts the FDB network. + Db = erlfdb:open(list_to_binary(ClusterFile)), + + %% Write a phase-specific key + Key = iolist_to_binary(["mvtest_", Phase]), + Val = iolist_to_binary(["value_", Phase]), + ok = erlfdb:set(Db, Key, Val), + + %% Read it back + case erlfdb:get(Db, Key) of + Val -> + io:format("[~s] Write+read OK: ~s = ~s~n", [Phase, Key, Val]); + Other -> + io:format(standard_error, "[~s] FAIL: expected ~p, got ~p~n", [Phase, Val, Other]), + halt(1) + end, + + %% After upgrade, verify the key written before upgrade is still readable + case Phase of + "post_upgrade" -> + case erlfdb:get(Db, <<"mvtest_pre_upgrade">>) of + <<"value_pre_upgrade">> -> + io:format("[~s] Cross-version read OK~n", [Phase]); + Other2 -> + io:format(standard_error, + "[~s] Cross-version read FAIL: expected ~p, got ~p~n", + [Phase, <<"value_pre_upgrade">>, Other2]), + halt(1) + end; + _ -> + ok + end, + + io:format("[~s] PASSED~n", [Phase]), + halt(0); + +main(_) -> + io:format(standard_error, "Usage: verify_fdb.escript ~n", []), + halt(1). From c78b1e2fa31d825c779f0351282b85db42a29dab Mon Sep 17 00:00:00 2001 From: Jesse Stimpson Date: Fri, 6 Mar 2026 09:31:59 -0500 Subject: [PATCH 3/4] Update changelog for v1.1.0 --- CHANGELOG.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index e49d285..3e58c5f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,16 @@ # Changelog +## v1.1.0 (2026-03-07) + +### Enhancements + + * Compile-time configuration: `rebar.config.script` reworked to support `ERLFDB_COMPILE_API_VERSION`, `ERLFDB_INCLUDE_DIR`, and `ERLFDB_FDBCLI` env vars, with better fallback behavior when fdbcli isn't available + * Multiversion CI job: new GitHub Action that compiles erlfdb against FDB 7.2.2, runs a single fdbserver, upgrades to 7.3.62, and verifies the multi-version client connects to both + +### Documentation + + * New `notes/configuration.md` covering all compile-time and runtime options + ## v1.0.0 (2026-02-28) No changes. From 37548e1404bd4a05f2a3a563ecb25f265e947a83 Mon Sep 17 00:00:00 2001 From: Jesse Stimpson Date: Fri, 6 Mar 2026 11:47:06 -0500 Subject: [PATCH 4/4] Improve documentation --- notes/configuration.md | 4 +++- rebar.config.script | 2 +- src/erlfdb_nif.erl | 1 + 3 files changed, 5 insertions(+), 2 deletions(-) diff --git a/notes/configuration.md b/notes/configuration.md index 216179f..c0a0896 100644 --- a/notes/configuration.md +++ b/notes/configuration.md @@ -18,6 +18,8 @@ export ERLFDB_INCLUDE_DIR=/opt/foundationdb/include When set, the build uses this value directly as the compile-time FDB API version, bypassing `fdbcli` checks. This is useful in environments where `fdbcli` is not installed on the build host but the FoundationDB client library and headers are available (e.g. multi-version support, cross-compilation or CI). erlfdb will compile only with features that are supported in this version, and will throw a `badarg` error when calling code executes an erlfdb function that is not supported. For example, calling `erlfdb:get_mapped_range/4` when compiled with version 710. +When unset, the build attempts to find the `fdbcli` binary to determine the compile-time API version (see below). + ```shell export ERLFDB_COMPILE_API_VERSION=730 ``` @@ -41,7 +43,7 @@ export ERLFDB_ASSERT_API_VERSION=0 ## Runtime Configuration -Runtime configuration is set via the `erlfdb` application environment, typically in a `sys.config` file or equivalent. +Runtime configuration is set via the `erlfdb` application environment, typically in a `sys.config` file or equivalent. Unless specified otherwise, all runtime configuration is evaluated when the NIF is loaded, and made permanent for the lifetime of the VM process. ### `api_version` diff --git a/rebar.config.script b/rebar.config.script index 4c76ab9..8968220 100644 --- a/rebar.config.script +++ b/rebar.config.script @@ -196,7 +196,7 @@ LogAssetsMap = fun ) end. -% If CompileAPIVersion undetected, print helpful informatiom about where to download the latest FDB Release +% If CompileAPIVersion undetected, print helpful information about where to download the latest FDB Release case CompileAPIVersion of undefined -> Log(info, "Checking for latest FoundationDB release...", []), diff --git a/src/erlfdb_nif.erl b/src/erlfdb_nif.erl index 41ce8e0..626042e 100644 --- a/src/erlfdb_nif.erl +++ b/src/erlfdb_nif.erl @@ -81,6 +81,7 @@ The NIF wrapper around all FoundationDB C API function calls. error_predicate/2 ]). +% This is defined at compile time using `erl_opts` in `rebar.config.script` -define(DEFAULT_API_VERSION, ?erlfdb_compile_time_api_version). -export_type([