CKKS bootstrapping incorrect under SPARSE_ENCAPSULATED on GPU (garbage output; SIGSEGV with numIterations=2); UNIFORM_TERNARY also affected

[davearcher]

First of all, thank you for FIDESlib — the GPU performance is remarkable (we measured bootstrapping at ~10 ms and deg-31 Chebyshev evaluation at ~7 ms at N=2^16, versus seconds on CPU), and the OpenFHE-interoperable design made it straightforward to try against an existing workload. We hit the issue below while running a bootstrapped CKKS workload (an encrypted transformer inference) that was designed and validated against stock OpenFHE 1.5.1, and we hope the detailed repro makes it easy to pin down.

Summary

With SPARSE_ENCAPSULATED secret keys, EvalBootstrap on GPU completes but returns results unrelated to the input (max error ≈ 1.0 on inputs in [-1, 1]), and EvalBootstrap(ct, 2, 17) crashes (SIGSEGV inside an OpenMP parallel region). UNIFORM_TERNARY bootstrapping also shows ~2^-1 error. SPARSE_TERNARY is correct (2^-9.8) with the identical harness and parameters, which we believe rules out a problem on our side.

Notably, the patched OpenFHE that deps/build.sh installs handles SPARSE_ENCAPSULATED correctly when driven directly through lbcrypto (2^-10.9 single-iteration, 2^-28.0 with numIterations=2), so the issue appears to live in the FIDESlib layer rather than in the patched OpenFHE.

Tested at main @ 786c760 ("Fix issue #31").

Measured matrix

Parameters: N = 2^16, depth 26, ScalingModSize 50, FirstModSize 51, FLEXIBLEAUTO, NumLargeDigits 4, HEStd_128_classic, 32768 slots, EvalBootstrapSetup({3,3}, {0,0}, 32768), input linspace[-1, 1] burned to level 21 with plaintext multiplies so the bootstrap is real (not the fresh-ciphertext no-op).

build	keydist	1-iter max err	2-iter (prec 17)
stock OpenFHE 1.5.1 (CPU)	SPARSE_ENCAPSULATED	4.7e-4 (2^-11.0)	5.1e-9 (2^-27.5)
patched OpenFHE from deps/build.sh, pure lbcrypto (CPU)	SPARSE_ENCAPSULATED	5.1e-4 (2^-10.9)	3.7e-9 (2^-28.0)
FIDESlib, GPU	SPARSE_ENCAPSULATED	~1.0 (output unrelated to input)	SIGSEGV
FIDESlib, GPU	SPARSE_TERNARY	1.2e-3 (2^-9.8) ✓	1.2e-3
FIDESlib, GPU	UNIFORM_TERNARY	5.0e-1 (2^-1.0)	4.2e-1

Two smaller observations we can split into separate issues if you prefer:

• The CPU fallback of EvalMult(const Ciphertext<DCRTPoly>&, Plaintext&) (api/CryptoContext.cpp:1220, taken when devices is empty) does std::any_cast<const lbcrypto::ConstPlaintext&>(pt->cpu), but MakeCKKSPackedPlaintext stores an lbcrypto::Plaintext; since std::any_cast requires an exact type match, this path always throws std::bad_any_cast.
• With SPARSE_TERNARY (where bootstrapping works), numIterations = 2 does not reduce the error (2^-9.76 → 2^-9.72), where stock OpenFHE improves 2^-11 → 2^-27.5 — iterated bootstrapping may not be wired through.

Environment

• GPU: NVIDIA GeForce RTX 5090 (32 GB), driver 580.159.04
• CUDA 13.0 (V13.0.88), sm_120
• Rocky/RHEL 9.8, GCC 14.2.1 (gcc-toolset-14), CMake 3.31.8
• FIDESlib main @ 786c760, built with -DFIDESLIB_INSTALL_OPENFHE=ON (its own patched OpenFHE), tests/benchmarks off
• Single device, SetDevices({0})

Reproduce

One self-contained source file, built once against FIDESlib and once against the patched OpenFHE alone as the control. Build commands and the full expected-output matrix are in the README of the attached sample; the core is:

cmake -S . -B build-fl -DUSE_FIDESLIB=ON \
      -Dfideslib_DIR=<prefix>/share/fideslib/cmake \
      -DOpenFHE_DIR=<patched-openfhe-prefix>/lib/OpenFHE
cmake --build build-fl -j
./build-fl/boot_precision_repro                    # encapsulated: garbage, then crash
KEYDIST=sparse  ./build-fl/boot_precision_repro    # correct (control)
KEYDIST=uniform ./build-fl/boot_precision_repro    # ~2^-1

boot_precision_repro.cpp (click to expand)

// Minimum reproducible sample: CKKS bootstrapping in FIDESlib returns
// garbage under SPARSE_ENCAPSULATED (max err ~1.0 one-iteration; crash
// with numIterations=2) and ~2^-1 under UNIFORM_TERNARY, while
// SPARSE_TERNARY is correct (2^-9.8) — and stock OpenFHE 1.5.1 handles
// SPARSE_ENCAPSULATED at 2^-11 / 2^-27.5 at identical parameters.
//
// Build twice from this one file (see CMakeLists.txt / README.md):
//   baseline:  links stock OpenFHE 1.5.1, namespace lbcrypto
//   fideslib:  -DUSE_FIDESLIB, links fideslib, namespace fideslib
//
// Optional env:
//   KEYDIST = encapsulated (default) | sparse | uniform
//
// Expected output: see the measured matrix in README.md.

#include <chrono>
#include <cmath>
#include <cstdlib>
#include <iostream>
#include <string>
#include <vector>

#ifdef USE_FIDESLIB
#include <fideslib.hpp>
using namespace fideslib;
#else
#include "openfhe.h"
using namespace lbcrypto;
#endif

static const uint32_t SLOTS = 32768;

int main() {
    CCParams<CryptoContextCKKSRNS> p;
    p.SetMultiplicativeDepth(26);
    p.SetScalingModSize(50);
    p.SetFirstModSize(51);
    p.SetScalingTechnique(FLEXIBLEAUTO);
    p.SetSecurityLevel(HEStd_128_classic);
    p.SetRingDim(65536);
    p.SetNumLargeDigits(4);
    p.SetBatchSize(SLOTS);

    const char* kd = std::getenv("KEYDIST");
    std::string k = kd ? kd : "encapsulated";
    p.SetSecretKeyDist(k == "sparse" ? SPARSE_TERNARY
                       : k == "uniform" ? UNIFORM_TERNARY
                                        : SPARSE_ENCAPSULATED);
    std::cout << "keydist: " << k << std::endl;

    auto cc = GenCryptoContext(p);
    cc->Enable(PKE);
    cc->Enable(KEYSWITCH);
    cc->Enable(LEVELEDSHE);
    cc->Enable(ADVANCEDSHE);
    cc->Enable(FHE);

    std::cout << "ctx ok" << std::endl;
    cc->EvalBootstrapSetup({3, 3}, {0, 0}, SLOTS);
    std::cout << "setup ok" << std::endl;
    auto keys = cc->KeyGen();
    std::cout << "keygen ok" << std::endl;
    cc->EvalMultKeyGen(keys.secretKey);
    cc->EvalBootstrapKeyGen(keys.secretKey, SLOTS);
    std::cout << "bootkeys ok" << std::endl;
#ifdef USE_FIDESLIB
    cc->SetDevices({0});                 // GPU 0; empty = broken CPU fallback
    cc->SetAutoLoadPlaintexts(true);
    cc->SetAutoLoadCiphertexts(true);
    cc->LoadContext(keys.publicKey);
#endif

    std::vector<double> xs(SLOTS);
    for (uint32_t i = 0; i < SLOTS; i++)
        xs[i] = -1.0 + 2.0 * i / (SLOTS - 1.0);
    auto pt_in = cc->MakeCKKSPackedPlaintext(xs);
    auto ct = cc->Encrypt(keys.publicKey, pt_in);
    std::cout << "encrypt ok" << std::endl;

    // Burn levels so the bootstrap is REAL (a fresh ciphertext is a no-op):
    auto ones = cc->MakeCKKSPackedPlaintext(std::vector<double>(SLOTS, 1.0));
#ifdef USE_FIDESLIB
    cc->LoadPlaintext(ones);
#endif
    auto burn = ct;
    while (burn->GetLevel() < 21) burn = cc->EvalMult(burn, ones);
    std::cout << "input burned to level " << burn->GetLevel() << std::endl;

    auto measure = [&](const char* tag, uint32_t iters, uint32_t prec) {
        try {
            auto b = burn;
            auto t0 = std::chrono::steady_clock::now();
            auto r = (iters == 1) ? cc->EvalBootstrap(b)
                                  : cc->EvalBootstrap(b, iters, prec);
            double ms = std::chrono::duration<double, std::milli>(
                            std::chrono::steady_clock::now() - t0).count();
            Plaintext pt;
            cc->Decrypt(keys.secretKey, r, &pt);
            pt->SetLength(SLOTS);
            auto got = pt->GetRealPackedValue();
            double worst = 0;
            for (uint32_t i = 0; i < SLOTS; i += 61)
                worst = std::max(worst, std::abs(got[i] - xs[i]));
            std::cout << tag << ": level " << burn->GetLevel() << " -> "
                      << r->GetLevel() << ", " << ms << " ms, max err "
                      << worst << " (~2^" << std::log2(worst) << ")"
                      << std::endl;
        } catch (const std::exception& e) {
            std::cout << tag << ": THROWS: " << e.what() << std::endl;
        }
    };
    measure("1-iter", 1, 0);
    measure("2-iter (precision 17)", 2, 17);
    return 0;
}

CMakeLists.txt

cmake_minimum_required(VERSION 3.16)
project(boot_precision_repro CXX)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_BUILD_TYPE Release)

# Two builds from one source file:
#   baseline: cmake -S . -B build-base -DOpenFHE_DIR=<stock 1.5.1>/lib/OpenFHE
#   fideslib: cmake -S . -B build-fl -DUSE_FIDESLIB=ON \
#               -Dfideslib_DIR=<prefix>/share/fideslib/cmake \
#               -DOpenFHE_DIR=<FIDESlib's patched OpenFHE>/lib/OpenFHE
option(USE_FIDESLIB "Link FIDESlib instead of stock OpenFHE" OFF)

find_package(OpenFHE REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${OpenFHE_CXX_FLAGS}")

add_executable(boot_precision_repro boot_precision_repro.cpp)
target_include_directories(boot_precision_repro PRIVATE
    ${OpenFHE_INCLUDE}
    ${OpenFHE_INCLUDE}/third-party/include
    ${OpenFHE_INCLUDE}/core
    ${OpenFHE_INCLUDE}/pke
    ${OpenFHE_INCLUDE}/binfhe)
target_link_libraries(boot_precision_repro PRIVATE ${OpenFHE_SHARED_LIBRARIES})

if(USE_FIDESLIB)
    find_package(fideslib REQUIRED CONFIG)
    target_compile_definitions(boot_precision_repro PRIVATE USE_FIDESLIB)
    target_link_libraries(boot_precision_repro PRIVATE fideslib::fideslib)
endif()

We're happy to test candidate fixes on this hardware, or to provide any further measurements that would help. Thanks again for the library!

[carlostriste]

Iterative bootstrap in GPU is not supported for now, let me look into the other failures, thank you so much.

[davearcher]

Thanks for the quick response, Carlos! Understood on iterated bootstrapping — those calls were incidental to the report, so please don't treat them as a request for that feature. The core issue for us is the single-iteration SPARSE_ENCAPSULATED result (output unrelated to the input), with the patched-oracle row in the table showing the expected behavior at the same parameters. Happy to run candidate fixes on the RTX 5090 environment whenever useful.

[davearcher]

A follow-up with three more findings, gathered while bringing a full bootstrapped CKKS workload (an encrypted BERT-tiny transformer inference, ~18k ops / 29 bootstraps per run) up on FIDESlib with SPARSE_TERNARY keys — which, once the items below were worked around, runs correctly on the RTX 5090: across a 40-sentence validation set, all 40 decisions match our plaintext reference implementation, at ~7.3 s per 4-sentence batch (the homomorphic portion is ~2.3 s; the rest is host-side plaintext encoding and device staging) versus ~470 s on CPU OpenFHE. The library's performance is genuinely spectacular; we hope these reports help polish the edges. Happy to split any of these into separate issues, and to test fixes on our hardware.

1. EvalChebyshevSeries returns garbage on mixed-sign asymmetric domains (possibly related to #31). At every degree we tested, domains with lo < 0 < hi and lo != -hi produce wildly wrong results, while symmetric and positive-only domains are exact:

degree	domain	max err vs Clenshaw
13	[-1, 1]	1.4e-9 ✓
13	[-0.9825, 0.0439]	1.5e+7
13	[0, 1.1]	3.0e-9 ✓
15	[-1, 1]	1.1e-9 ✓
15	[-0.9825, 0.0439]	3.5e+8
15	[0, 1.1]	1.9e-9 ✓

We also have a same-input, same-op side-by-side from the application trace: identical operands (max ~1e-4, well inside the domain) produce 0.839 through lbcrypto and -0.059 / +0.25 / -1.52 through the GPU path. Workaround that costs nothing, in case it helps others: shift the operand by the domain center (an addition, level-free) and evaluate on the centered symmetric domain with the identically-refit polynomial.

2. There is no public device-to-host download except through Decrypt. CiphertextImpl::EnsureLazyCPUCopy() is a host-side copy-on-write only, so after GPU computation the cpu member holds stale data unless one decrypts. For a server that must serialize a result it cannot decrypt (the standard client/server FHE deployment), we had to replicate the internals of Decrypt's download (GetDeviceCiphertext → store(raw) → GetOpenFHECipherText, with a template ciphertext when the host copy lacks towers). A public SynchronizeToHost(ct) — essentially that code path minus the decryption — would make the server use case first-class.

3. Key receivers need two undocumented manual declarations. For a process that deserializes keys rather than generating them (again, the client/server split): DeserializeEvalAutomorphismKey does not populate rotation_indexes, and nothing except EvalBootstrapKeyGen (a secret-key call the server can't make) populates slots_bootstrap — with either missing, LoadContext silently stages nothing for that category and the first dependent op throws out_of_range from GetRotationKey. Assigning both public members before LoadContext works; having deserialization record what it loaded (or receiver-side declaration APIs) would remove the trap. Relatedly, keys must be generated against the patched OpenFHE your deps/build.sh installs — stock OpenFHE 1.5.1 produces bootstrap automorphism indices the patched EvalBootstrapSetup doesn't request, which surfaces as map::at during staging; a docs note would save the next integrator a puzzle.

4. Observation: in-circuit noise runs ~6x higher than lbcrypto at identical parameters. Over the full 18k-op circuit our decrypted outputs deviate from the plaintext model by mean 0.30 / max 0.67 (in units where lbcrypto at the same parameters gives mean 0.012 / max 0.026). We verified this is in-compute rather than in our readback (the facade's own Decrypt of the final ciphertext matches our serialized result to 2e-3), and isolated single ops (rotation, multiply, bootstrap, Chebyshev on well-formed domains) all measure clean — so it appears to be accumulated rescale/key-switch rounding across long chains. Not blocking for us (decisions are stable), but worth knowing whether it matches your expectations for the engine's precision profile.

[seyfal]

Reproduced on an H200 (sm_90, CUDA 13.0, driver 580) at main 786c760 with your program unchanged: uniform 2^-1.25, sparse 2^-9.86, encapsulated max error 1.0. So it is not specific to the RTX 5090.

Going through the low-level API instead (GetRawParams / GenCryptoContextGPU / AddBootstrapPrecomputation / Bootstrap), the failure follows the bootstrap configuration rather than the key. With the same UNIFORM_TERNARY key at N = 2^16, 32768 slots and our parameters (scale 52, first 56, dnum 3, depth 25): FIDESlib::ENCAPS gives max error 2^-7.4, FIDESlib::SPARSE 2^-6.9, FIDESlib::UNIFORM 2^-2.7 (K = 512, degree-88 EvalMod, 6 double-angle iterations). With your parameters (50 / 51 / dnum 4) UNIFORM gives 2^-0.3 and ENCAPS 2^-5.4. SPARSE_TERNARY keys with the SPARSE configuration are fine (2^-9.1 with your parameters, 2^-11.8 with ours). As far as we can tell the compat header picks the UNIFORM configuration for a uniform key, so that path is where we would look; the ENCAPS configuration handles the same key correctly. One more data point: SPARSE_ENCAPSULATED keys throw std::out_of_range (vector size 1) in the low-level path.

Repro program, with environment variables for the key distribution, bootstrap configuration, slot count and parameters: https://github.com/seyfal/FIDESlib/blob/repro/issue33/repro-issue33/repro33.cu. Happy to run anything else that helps.

Fable 5.1 on behalf of Seyfal