refactor(rpc/get_account): batch storage map proofs

[kkovaacs]

Summary

GetAccount now returns a PartialSmt with all requested key-value pairs.

• A request for explicit map keys always returns one partial SMT covering those keys.
• Partial SMTs are scoped to one storage map at one block. Proofs from different maps or roots are never merged.
• The response carries the original, unhashed StorageMapKeys. The SMT contains hashed keys, so the raw keys cannot be recovered from the tree.
• Values are not duplicated outside the SMT. A client obtains a value by hashing the raw key and calling PartialSmt::get_value().
• too_many_entries, all_entries, and partial_map become mutually exclusive results.
• The removed entries_with_proofs field number and name are reserved. Even though compatibility is intentionally broken, reusing the field could make an old client try to decode the new length-delimited message as the old one.
• The compact protobuf representation mirrors miden_crypto::merkle::smt::UniqueNodes. Do not encode PartialSmt as an opaque byte string.

Closes #617

Changelog

[[entry]]
scope       = "rpc"
impact      = "breaking"
description = "Storage map proofs for multiple keys in `AccountResponse` are now represented as a batched `PartialSmt`."

[kkovaacs]

Looks like there is a bug in PartialSmt::from_unique_nodes() that might prevent us from taking full advantage of this optimization: 0xMiden/miden-vm#3470

[bobbinth]

Looks like there is a bug in PartialSmt::from_unique_nodes() that might prevent us from taking full advantage of this optimization: 0xMiden/miden-vm#3470

Once 0xMiden/miden-vm#3471 is merged, we should be able to do a patch release and it'll flow through to here immediately.

[kkovaacs]

@igamigo Do you have an estimate on how many storage keys a client usually requests from a storage map in a single GetAccount request?

As far as I understand there are two cases where the client is requesting specific keys from a storage map:

• When preparing transactions involving public foreign accounts: the client fetches all keys explicitly referenced in AccountStorageRequirements when adding the foreign account(s) to the TransactionRequestBuilder.
• Lazily during transaction execution: when a required storage-map witness is absent locally and from the in-memory cache.

As far as I understand in the lazy (execution) use case the client is always requesting a single key from a single storage slot. There is nothing to be gained in that case from the optimization this PR implements.

In the first case though: how much this optimization helps depends on the number of the storage map keys typically required. Do we have an estimate on how many keys we're expecting there?

CC @Mirko-von-Leipzig

[igamigo]

@igamigo Do you have an estimate on how many storage keys a client usually requests from a storage map in a single GetAccount request?

Not really, and as we discussed the other day, this is very transaction/smart contract-specific. It's very hard to generalize so I would not let any specific usecase inform any limits here. For example, I think Zoro is the only user who has hit the 64 key limit before, and they worked around it by sharding data in different accounts AFAIK. They were going to try locally following the foreign account as an alternative, but surely they would benefit from a higher limit.

Asset callbacks will probably be the dominant FPI usecase but at first sight I don't think you would need to fetch a large amount of keys there on average (not sure though).