Observer `get-burn-proof` API Returns Null Despite Burn Being Processed

[freesig]

Observer get-burn-proof API Returns Null Despite Burn Being Processed

Summary

The /get-burn-proof/{owner}/{nonce} API endpoint returns null on the observer node even when:

1. The balance has been reduced (indicating the burn was processed)
2. The GPU publisher has generated the ZK proof
3. Network traffic shows the observer received data from the publisher

Root Cause

The get_burn_proof function only reads from the merkle trie, while get_balance reads from uncommitted state first.

Technical Details

The observer processes state transitions which write to recording.writes (a pending buffer). These writes are only moved to the merkle trie when flush() is called. However, flush() is only called inside Witness::try_from_storage() (part of the ProofConversions trait), which appears to not be invoked for observers.

Inconsistent Read Patterns

Function	Checks recording.writes?	Checks tree?	Location
get_balance	Yes	Yes	app.rs:442
is_burned	Yes	Yes	app.rs:557
get_burn_proof	No	Yes	state_reads.rs:139

Code Flow

get_balance (works correctly):

// app.rs:442-450
if let Some(delta) = self.recording.writes.get(&balance_key) {
    // Returns from uncommitted state
}
// Falls back to trie

get_burn_proof (broken):

// state_reads.rs:139
let value = match mem.app.tree.get(&key) {  // ONLY reads from trie
    Ok(Some(v)) => v,
    Ok(None) => return Ok(None),  // Returns None if not flushed
    ...
};

Why flush() Isn't Called on Observers

The flush() method is defined in app.rs:585 and is only called from Witness::try_from_storage() at app.rs:662:

fn try_from_storage(store) -> Result<Self> {
    match store {
        DataStorage::Memory(app) => {
            // Generate proof from app's trie
            let proof = MultiProofBuilder::new(app.tree.storage(), app.tree.root_hash())
                .add_keys(app.recording.keys())
                .build()?;
            // Flush recording
            app.flush();  // <-- Only place flush() is called
            Ok(Witness { proof })
        }
    }
}

This method is part of the ProofConversions trait and is called by the void-app-node framework. For publishers, this is called after each block to generate the ZK proof witness. For observers, if the framework doesn't call this (because observers verify proofs rather than generate them), then flush() is never invoked.

Impact

• Users cannot retrieve burn proofs to complete withdrawals on-chain
• The API returns null instead of an error, making debugging difficult
• Balance queries work correctly (misleading users into thinking the withdrawal succeeded)

Evidence

1. Live endpoint returns null: https://orderbook-api.bigbangblock.builders/get-burn-proof/00a568abd68d9cb4b5acbd87d7bd04fd95d6bcb4e6/5
2. Tests confirm the pattern: app/tests.rs:163-167 shows that is_burned returns true before flush (from recording), and the trie query only works after flush
3. ALB configuration confirms routing: alb.tf:182-200 routes /get-burn-proof/*/* to the observer target group

Potential Solutions

Option 1: Fix in void-app-node (Proper Fix)

Ensure the framework calls flush() (or try_from_storage) for observers after each block, not just for publishers.

Pros: Fixes the root cause
Cons: Requires changes to external framework

Option 2: Add flush() to api_update

Add a flush() call to the api_update function since it runs after state_transition_function and has mutable access to state.

Pros: Application-level fix
Cons: May interfere with proof generation if framework expects unflushed state; needs careful coordination with void-app-node

Option 3: Make get_burn_proof Check recording.writes

Modify get_burn_proof to check recording.writes first and return a meaningful error if the burn is pending but not flushed.

// Check if burn exists in trie
let value = match mem.app.tree.get(&key) {
    Ok(Some(v)) => v,
    Ok(None) => {
        // Check if pending in recording.writes
        if mem.app.recording.writes.contains_key(&key) {
            return Err(anyhow::anyhow!(
                "Burn exists but proof not yet available - state pending flush"
            ));
        }
        return Ok(None);
    }
    ...
};

Pros: Provides better error feedback; no changes to flush timing
Cons: Doesn't fix the underlying issue; users must retry

Option 4: Separate Read Path for Observers

Create an observer-specific API endpoint or mode that reads from recording.writes and returns burn data without a merkle proof (for informational purposes only).

Pros: Works around the issue
Cons: Two different APIs; non-standard behavior

void-app-node Framework Analysis

Publisher vs Observer State Transition Functions

The framework has two separate functions for state transitions in void-examples/node/src/lib.rs:

Publisher: state_transition_with_storage (lines 569-607)

async fn state_transition_with_storage<S, P, Stf, ApiU>(
    block: Block,
    storage: &S,
    stf: Arc<Stf>,
    api_update: Arc<ApiU>,
) -> anyhow::Result<(Block, Height<P>)>
{
    // ...
    Storage::Memory(mem) => mem.apply(|apply| {
        stf.apply(&block, DataStorage::Memory(apply.app))?;
        apply.latest_header.update_latest_block(...)?;
        let proof = P::try_from(DataStorage::Memory(apply.app))?;  // <-- CALLS try_from which flushes
        api_update.apply(&block, ...)?;
        Ok((block, Height::new(height, proof)))
    }),
}

Observer: observer_state_transition_with_storage (lines 609-642)

async fn observer_state_transition_with_storage<S, Stf, ApiU>(
    block: Block,
    storage: &S,
    stf: Arc<Stf>,
    api_update: Arc<ApiU>,
) -> anyhow::Result<()>  // Note: returns () not (Block, Height<P>)
{
    // ...
    Storage::Memory(mem) => mem.apply(|apply| {
        stf.apply(&block, DataStorage::Memory(apply.app))?;
        apply.latest_header.update_latest_block(...)?;
        api_update.apply(&block, ...)?;
        Ok(())  // <-- NO proof generation, NO flush!
    }),
}

Key Difference: Observer function does not have P as a type parameter and never calls P::try_from(), which means ProofConversions::try_from_storage() is never invoked, and therefore flush() is never called.

Why Observers Don't Generate Proofs

This is intentional - observers verify proofs received from publishers rather than generating their own. The proof type P is not even available in the observer function signature.

Recommended Fix in void-app-node

Option A: Add AppFlush Trait (Cleanest)

Add a new trait that allows flushing without proof generation:

// In void-app-node/src/transitions.rs
pub trait AppFlush {
    fn flush(&mut self);
}

Then modify observer_state_transition_with_storage:

async fn observer_state_transition_with_storage<S, Stf, ApiU>(
    block: Block,
    storage: &S,
    stf: Arc<Stf>,
    api_update: Arc<ApiU>,
) -> anyhow::Result<()>
where
    S: Store,
    S::App: AppFlush,  // NEW: require App to implement AppFlush
    Stf: AppTransition<S::App, S::DbData>,
    ApiU: ApiTransition<S::App, S::Api, S::DbData>,
{
    match storage.storage() {
        // ... Db branch unchanged ...
        Storage::Memory(mem) => mem.apply(|apply| {
            stf.apply(&block, DataStorage::Memory(apply.app))?;
            apply.latest_header.update_latest_block(...)?;
            api_update.apply(&block, ...)?;
            apply.app.flush();  // NEW: flush after api_update
            Ok(())
        }),
    }
}

Application side (orderbook):

// In crates/orderbook/src/app.rs
impl void_app_node::transitions::AppFlush for App {
    fn flush(&mut self) {
        App::flush(self);  // Reuse existing flush method
    }
}

Pros:

• Clean separation of concerns
• No proof generation overhead
• Application controls flush implementation
• Framework explicitly supports the pattern

Cons:

• Requires adding trait bound to observer function
• Applications must implement the trait

Option B: Add flush_storage to ProofConversions Trait

Extend the existing trait with an optional flush method:

// In void-app-node/src/proof.rs
pub trait ProofConversions {
    type App;
    type DbData;

    // Existing methods...
    fn digest(&self) -> [u8; 32];
    fn into_bytes(self) -> Vec<u8>;
    fn try_from_bytes(bytes: &[u8]) -> anyhow::Result<Self> where Self: Sized;
    fn try_from_storage(store: DataStorage<'_, '_, Self::App, Self::DbData>) -> anyhow::Result<Self> where Self: Sized;

    // NEW: Flush storage without generating proof
    fn flush_storage(store: DataStorage<'_, '_, Self::App, Self::DbData>) {
        // Default implementation calls try_from_storage and discards result
        let _ = Self::try_from_storage(store);
    }
}

Then modify observer_state_transition_with_storage to call P::flush_storage() where P is passed as a type parameter.

Pros:

• Uses existing trait infrastructure
• Applications can override with optimized implementation

Cons:

• Default implementation wastes CPU on proof generation
• Requires adding P type parameter to observer function

Option C: Post-Block Flush Hook

Add a hook in the observer stream processing:

// In run_observer_stream() at line 864-910
oracle_stream
    .block_height_parent(last_parent_height, last_parent_hash)
    .blocks_await_proofs(proof_heights_stream)
    .and_then(|block| {
        observer_state_transition_with_storage(block, &node.storage, stf.clone(), api_update.clone())
    })
    .and_then(|_| {
        // NEW: Flush after each block
        node.storage.flush_app()
    })
    .push_notification(node.state_notification.clone())
    .try_for_each(|_| std::future::ready(Ok(())))
    .await?;

This requires adding a flush_app() method to the Storage trait.

Recommended Approach

1. Immediate fix: Implement Option A (AppFlush trait) in void-app-node
2. Application change: Implement AppFlush for App in orderbook

This is the cleanest solution because:

• It's explicit about what observers need
• No wasted proof computation
• Applications control their flush logic
• The trait bound makes the requirement visible in the type system

Files to Modify

void-app-node (framework)

• src/transitions.rs - Add AppFlush trait
• src/lib.rs - Add S::App: AppFlush bound to observer_state_transition_with_storage and call apply.app.flush()

orderbook (application)

• crates/orderbook/src/app.rs - Implement AppFlush for App

Related Code References

void-app-node

• observer_state_transition_with_storage: lib.rs:609-642
• state_transition_with_storage (publisher): lib.rs:569-607
• Memory::apply: memory.rs:105-116
• ProofConversions trait: proof.rs:32-45

orderbook

• flush() definition: app.rs:585-598
• try_from_storage (calls flush): app.rs:648-666
• get_burn_proof (only reads trie): state_reads.rs:122-183
• get_balance (reads recording first): app.rs:428-456
• is_burned (reads recording first): app.rs:553-568