Add HTTP/2 support with dynamic table synchronization - #60
Merged
Conversation
…amic table Reverts commit 3743a34, resolving the conflict in example/src/server.bpf.c against the newer BEELINE_* macro refactor (the h1/h2 parser stub functions now use those macros instead of hand-written __noinline stubs). Also extends beeline's h2 parser with a way to read back its mirrored dynamic table: `h2_frame` now reports the table's live entry count, and a new `BEELINE_H2_GET_DT_ENTRY`/`get_dt_entry` freplace function (wired through `h2::Parser::replace_get_dt_entry`) lets a target program read an entry by its HPACK index. This is the foundation for keeping the example fast path's eBPF-side dynamic table in sync with user space's. Note: this was pushed via the GitHub API without a local build/test pass -- the container's Bash execution broke (a stray `mount --bind` onto /sys/fs/cgroup while trying to work around a cgroup v1/v2 mismatch in the test harness shadowed the hierarchy the sandbox's own exec wrapper depends on). Commit 1 (the revert) was previously verified to build and this content is unchanged from that; commit 2's additions compile as plain C/Rust additions to working code but have not been built or run through the BPF verifier.
Adds `h2_frame.dt_count` (the table's live entry count as of the frame just parsed) and a new `BEELINE_H2_GET_DT_ENTRY`/`get_dt_entry` freplace function, wired through `h2::Parser::replace_get_dt_entry`, that reads a dynamic table entry back out by its HPACK index (reusing `_get_table_entry`, the same lookup `extract_match` already relies on). `header_field` moves into beeline.h so both the parser and a target program that reads entries agree on its layout. This lets a target program that can answer some HTTP/2 requests without forwarding them to user space (like the example fast path) find out whether, and with what, the dynamic table it shares with user space has diverged. Unbuilt/unverified -- see the previous commit's message for why.
The previous commit dropped the `__sink(ret)` from the `BEELINE_EXTRACT_MATCH` stub. Without it clang can see that `ret` is always -1 and folds it away, so the freplace attach point loses the return value and every `extract_match` call reports "no match", breaking all seven capturing HTTP/1.1 tests. Also realigns the line continuations of the new `BEELINE_H2_GET_DT_ENTRY` macro with the rest of the file.
A request the fast path answers itself never reaches user space, but its HEADERS block may still have added entries to the client's dynamic table. The mirrored table in eBPF picks those up, user space's does not, so every later index the client sends resolves against a table that is missing entries -- and HPACK never recovers from that. The fast path now tracks, per connection, how many entries user space has been told about (`dt_synced`). Answering a request itself leaves that number behind while the mirrored table grows; the difference is the lag. On the next message it forwards, it replays the missing entries as an HPACK block of literal-with-incremental-indexing fields, oldest first, wrapped in a frame of the unassigned type 0xFB and prepended to the message. A decoder that runs the block ends up with exactly the entries the fast path saw, in the order it saw them. A lag too large for one sync frame stops the fast path from answering on that connection until user space has caught up, and a sync frame that cannot be built drops the connection rather than letting user space decode against a table it can no longer trust. `h2_frame` now reports the table's entry count from before the frame as well, so the lag can be computed without counting the entries of the very message being forwarded -- user space adds those itself.
Adds a TcpListener wrapper in the shape of tokio_rustls: the listener hands back a stream that speaks a small protocol of its own underneath the one the server speaks. The fast path's sync frames can show up at any frame boundary, not just at the start of a connection -- one is prepended whenever the fast path forwards a message after having answered something itself -- so the stream follows the HTTP/2 framing rather than peeking once at the handshake. Sync frames are taken out and handed to a `SyncHandle`, which the connection loop drains to prime its decoder; everything else is passed through, and a connection that does not open with the HTTP/2 preface is left alone entirely. Covered by unit tests for the framing (including a frame split one byte per read, several updates in a row, an empty update and an HTTP/1.1 connection) and a round trip through a real socket.
Completes the handover on the user space side. The dynamic table of an HTTP/2 connection lives in h2's HPACK decoder, and neither h2 nor hyper hands it out, so `vendor/h2` is h2 0.4.16 with an accessor chain bolted on down to `Decoder::prime`, which runs a header block for its effect on the table and drops the fields. The workspace points at it through [patch.crates-io]. Every addition is marked BEELINE PATCH; it is a hack for this example and nothing else. The connection loop drives h2 directly rather than going through `axum::serve`, since owning the `h2::server::Connection` is what makes that accessor reachable, and applies any pending update before each poll of the connection. The application stays an ordinary axum Router. HTTP/1.1 connections skip all of this and go to hyper as before -- they spell their headers out, so nothing can get out of step. Ordering is the subtle part: the fast path prepends its update to the very message carrying the request that needs it, so both arrive in one read. The stream therefore stops handing out bytes at an update and only resumes once it has been taken off the handle, which is what stops the codec from decoding that request against a table it has not yet caught up. Covered by integration tests over real sockets: a request whose header is a dynamic table index resolves with the update and fails without it, both against a bare h2 server and through the full connection loop, plus HTTP/1.1 and unaffected HTTP/2 requests. The header blocks the tests build are the ones `render_dt_sync` emits, so the wire format is checked against a real HPACK decoder.
Sending only the entries added since user space last looked is wrong as soon as a request the fast path answers evicts something: user space is then holding entries the client has already dropped, and no set of additions puts the two back in step. Worse, the entry count it was tracking goes down rather than up, so the old code computed a lag of zero and sent nothing at all -- exactly the case that needed fixing. The fast path now keeps a flag per connection rather than a count, and the sync frame carries the whole table: an HPACK size update to zero and back to the peer's `SETTINGS_HEADER_TABLE_SIZE` (which `h2_frame` now reports), which empties the receiving decoder's table, followed by every live entry replayed oldest first. Wherever the two tables had got to, the receiver ends up holding what the fast path mirrors. A table too large for one frame stops the fast path from answering on that connection, which is always safe: forwarding the request is what keeps user space in step in the first place. Covered by a test that hands a server a stale table and then a fresh one, and checks the client's index resolves against the fresh table alone.
The dynamic table handover failed on ordinary requests with "unable to maintain the header compression context". HPACK lets a peer choose per string whether to Huffman code it, and curl spells `accept: */*` out because coding it saves nothing -- then indexes it. The parser stored those bytes without recording which form they were in, and the fast path copied them back out with the H bit set regardless, so the server tried to read `*/*` as a Huffman code and the block failed to decode. `hdr_match` and `header_field` now carry that flag, taken from the top bit of a string's length prefix, and the sync frame sets the H bit from it per string. It is captured inside `_parse_hpack`, in a branch that already exists, rather than as a test of its own in the parsing loop: the obvious spelling pushed the loop past the verifier's 8192 jump sequence limit, while this leaves its state count within a few percent of what it was. The same flag fixes the entry sizing, which had the same assumption baked in: RFC 7541 sizes an entry by its name and value as text, so a Huffman coded string counts for what it decodes to and one that was spelled out counts for its own length. Sizing a raw string as though it were coded gave the mirrored table a size the client never had, and so evicted at the wrong point. Covered by tests that hand over a value that was not Huffman coded, and a mix of both, through the bare decoder and the full connection loop. Reintroducing the old behaviour fails them.
HPACK lets a peer spell a string out instead of Huffman coding it, and real clients do -- curl sends `accept: */*` that way, because coding it saves nothing. Nothing covered that: `h2`'s encoder always codes, so the client these tests use cannot produce one, which is why the parser reading every stored string as Huffman went unnoticed until the fast path replayed one. Adds a client that writes its own HPACK, and two tests over it. The first checks that a spelled out value is stored and captured as sent, and that the table is sized by name and value as text: sizing a spelled out string as though it were coded gives the mirror a size the client never had, so it evicts at the wrong point. The second adds such an entry and then refers to it by index alone on a later request, which is what a client does once the entry is in its table. Both blocks were checked against a real HTTP/2 server first, so a failure means the parser disagrees with it rather than that the test built bad HPACK.
Nothing covered what the parser does with input a peer would reject, and it walks whatever it is handed. Adds four cases, each asserting that nothing was captured and that the dynamic table was left alone: a frame whose header claims bytes that were never sent, a field indexed past the end of the table, and a value whose length reaches past the frame it sits in. The fourth sends a frame of an unassigned type, which RFC 7540 says has to be discarded rather than choked on, and checks the parser picks the stream back up on the frame after it. The table assertions are the ones worth having: a malformed field that is merely not captured can still be added to the table, and an entry the client never made puts every later index out by one. `send_raw` deliberately reads nothing back. A read of whatever has arrived can stop mid frame and leave the stream out of step for the next one, and it is not needed: an `sk_msg` program runs as part of the send, so the parser has seen the bytes by the time the write returns. The premises were checked against a real HTTP/2 server first: that it keeps serving across the unknown frame, so the recovery case is really answered, and that it rejects the two malformed blocks, so they are invalid rather than accidentally well formed.
The test expected nothing to be captured, and CI came back holding the HTTP/2 connection preface. Match id 0 is written by the h1 parser when it matches the preface and upgrades the connection, and the test program leaves the captures alone when a parse fails rather than clearing them, so the id still held what the upgrade put there. Nothing captured was the wrong thing to look for. What the frame must not do is change anything, so the test now gets a request through first and checks that the malformed frame afterwards leaves both the capture and the table exactly as they were.
Adds --with-ebpf. Without it nothing is loaded into the kernel at all and the same server runs as an ordinary axum application, which is both the thing worth comparing against and the only way to run the example on a kernel whose verifier will not take the HTTP/2 parser. The assets are now a page rather than two lines: index.html references a stylesheet, a script and three images, and the script reloads each of them and reports what came back and how long it took, which is where the difference the fast path makes shows up. Only the three text assets fit its routing table, so the limit goes up to 16K to make room for the script, and a response too large for it is now declined with a note rather than refusing to start. The images are far past any workable limit and are meant to be: a path the fast path does not know is one it passes on, which is exactly what should happen to them. Handing it every asset and letting it say which it can take beats choosing for it.
lerboe
commented
Aug 20, 2026
| u16 idx; | ||
| u16 len; | ||
| bool in_msg; | ||
|
|
Owner
Author
There was a problem hiding this comment.
Reduce inline comments to a minimum. This one is not necessary.
| u32 sid; | ||
| u8 type; | ||
| u8 flags; | ||
|
|
Owner
Author
There was a problem hiding this comment.
Mkae these comments more concise.
| .idx = k, | ||
| .len = HEADER_FIELD_MASK, | ||
| .in_msg = false, | ||
| // the entry carries its own encoding, which |
| u16 s = s_any; | ||
| struct msg_ctx ctx = _new_msg_ctx(msg); | ||
|
|
||
| // the entry count is reported on either side of the decode, so that a |
| return res; | ||
| } | ||
|
|
||
| // Reads the `idx`th entry of `conn`'s dynamic table into `out`, the same way |
| /// # Arguments | ||
| /// | ||
| /// * `get_dt_entry_fn` - The name of the dynamic table entry reader function in the target program | ||
| pub fn replace_get_dt_entry<S: ToString>(mut self, get_dt_entry_fn: S) -> Parser { |
Owner
Author
There was a problem hiding this comment.
For public functions, spell dynamic table out, so replace "dt"
| } | ||
| } | ||
|
|
||
| pub struct OpenObject { |
Owner
Author
There was a problem hiding this comment.
You should use xbpf::OpenObject instead
| clap = { version = "4", features = ["derive"] } | ||
| # the connection loop drives h2 itself so that it can reach the dynamic table, | ||
| # see the [patch.crates-io] section of the workspace manifest | ||
| h2 = "0.4.16" |
Owner
Author
There was a problem hiding this comment.
I pushed the h2 patch to https://github.com/lbrndnr/h2. Use this instead of vendoring the whole repo.
Takes the h2 patch from https://github.com/lbrndnr/h2 rather than carrying a copy of the whole crate, which drops 22k lines from the diff. The fork is v0.4.17 with the same patches, checked to still carry every accessor this depends on before the switch. `example::fastpath::OpenObject` was a byte for byte copy of `xbpf::OpenObject`, so it is gone and the example uses that one. `replace_get_dt_entry` becomes `replace_get_dynamic_table_entry`, and the comments the review pointed at are cut back or dropped: the encoding of a `hdr_match` moves into the struct's own doc comment instead of sitting beside the field, and what is left on the frame counts, the table pointer and `get_dt_entry` says only what the code does not.
Owner
Author
|
Closes #9 |
lerboe
force-pushed
the
claude/http2-dynamic-table-sync-nfukrl
branch
from
August 20, 2026 11:52
8f19d58 to
ccde7f2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds comprehensive HTTP/2 protocol support to the project, including a complete implementation of the h2 library and integration with the fast path's dynamic table synchronization mechanism. The changes enable the server to handle HTTP/2 connections and properly replay HPACK dynamic table updates that the eBPF fast path applied without the server's knowledge.
Key Changes
Added h2 HTTP/2 library: Vendored the complete h2 crate (v0.4.16) with full HTTP/2 client and server implementations, including:
Implemented dynamic table synchronization:
DT_SYNC_FRAME_TYPEsupport in the eBPF parser to extract HPACK dynamic table entries that the fast path addedBeelineListenerwrapper that intercepts and extracts dynamic table sync frames from the byte stream before they reach the h2 codecSyncHandlemechanism to pass extracted table entries to the connection loopIntegrated h2 into the server connection loop:
h2servemodule that drives the h2 connection directly (rather than through axum) to access the HPACK decoder's dynamic tableBeelineListenerandh2servefor HTTP/2 connection handlingserve.rs,dt_sync.rs) to validate the dynamic table handover without eBPFUpdated eBPF components:
server.bpf.cwithupgraded_connsmap to track HTTP/2 connectionsbeeline.hto include dynamic table state tracking in frame metadataAdded library exports: Created
example/src/lib.rsto expose server components for testingNotable Implementation Details
DT_SYNC_FRAMEentries to the next forwarded message, which theBeelineListenerextracts and applies to the h2 decoder before normal frame processinghttps://claude.ai/code/session_01GSbSGj4FYh4M6TcXgnsiqN