Right now the backend sees the proxy's address and nothing else. create_connect() opens the socket and the first byte through it is the client's first byte, so there is no point at which the client's address could be carried. For anything that authorizes, rate limits or audits by source address that makes us unusable, and it is the single thing most likely to end an evaluation.
The relevant detail is that this is not just a missing feature, it is the key to the one distribution channel that matters. man mosquitto.conf documents enable_proxy_protocol [ 2 | 1 ] and says of it:
This option requires the use of a load balancer/proxy such as HAProxy in front of
That sentence is the position to take. Mosquitto is the most deployed broker there is, its documentation names the proxy to put in front of it, and we are not that proxy because we cannot speak the protocol it needs. EMQX reads it too. NanoMQ does not, so it is not a target for this.
What to build
proxy-protocol: per listener, taking off (default), v1 or v2, emitted as the first bytes on the backend socket once it connects and before anything the client sent.
- v1 is the text form:
PROXY TCP4 <src> <dst> <sport> <dport>\r\n, and TCP6 for the v6 case. Easy to get right and easy to read in a tcpdump, which is why it is worth having even though v2 is better.
- v2 is the binary form, and is the one to default to in the documentation, since it is the only one that can carry TLS information later. The 12-byte signature, then version and command, then family and protocol, then a two-byte length, then the address block.
- The address written is the client's, which
handle_accept() already copies into proxy->client_addr and proxy->client_addrlen, so the data is present and this is a question of where to put the write rather than of where to get the address.
The header has to be written before any client bytes reach the backend. The proxy already has the state for that: the backend leg is in PS_SERVER_CONNECTING until proxy_handle_connect() completes, and client data that arrives in the meantime is queued in c2s. Enqueueing the header onto the front of c2s at connect time is one option; writing it directly on the connect completion is another. Whichever, the failure mode to avoid is a partial header write on a non-blocking socket being treated as done, since the backend will then read a truncated signature and, correctly, drop the connection.
Acceptance
- v1 and v2, IPv4 and IPv6, configurable per listener and off by default, since a backend that is not expecting a header will treat it as protocol garbage.
- The header survives a partial write.
- An integration test that runs Mosquitto with
enable_proxy_protocol 2 on a plaintext listener behind us, connects a client, and asserts that the broker's log records the client's address rather than ours. Mosquitto is worth the test container rather than a hand-rolled parser, since the point of the exercise is that a real broker accepts what we emit.
example/default.yml and README.md updated, and the Mosquitto stanza that goes with it written out in full, because the thing being sold here is that it works with Mosquitto rather than that the bytes are correct.
Notes
hitch has had this since well before 1.7.0, with write-proxy-v1, write-proxy-v2, write-ip and proxy-proxy for forwarding a header that arrived from further upstream. proxy-proxy is out of scope here; the outbound direction is what unblocks a deployment.
Right now the backend sees the proxy's address and nothing else.
create_connect()opens the socket and the first byte through it is the client's first byte, so there is no point at which the client's address could be carried. For anything that authorizes, rate limits or audits by source address that makes us unusable, and it is the single thing most likely to end an evaluation.The relevant detail is that this is not just a missing feature, it is the key to the one distribution channel that matters.
man mosquitto.confdocumentsenable_proxy_protocol [ 2 | 1 ]and says of it:That sentence is the position to take. Mosquitto is the most deployed broker there is, its documentation names the proxy to put in front of it, and we are not that proxy because we cannot speak the protocol it needs. EMQX reads it too. NanoMQ does not, so it is not a target for this.
What to build
proxy-protocol:per listener, takingoff(default),v1orv2, emitted as the first bytes on the backend socket once it connects and before anything the client sent.PROXY TCP4 <src> <dst> <sport> <dport>\r\n, andTCP6for the v6 case. Easy to get right and easy to read in a tcpdump, which is why it is worth having even though v2 is better.handle_accept()already copies intoproxy->client_addrandproxy->client_addrlen, so the data is present and this is a question of where to put the write rather than of where to get the address.The header has to be written before any client bytes reach the backend. The proxy already has the state for that: the backend leg is in
PS_SERVER_CONNECTINGuntilproxy_handle_connect()completes, and client data that arrives in the meantime is queued inc2s. Enqueueing the header onto the front ofc2sat connect time is one option; writing it directly on the connect completion is another. Whichever, the failure mode to avoid is a partial header write on a non-blocking socket being treated as done, since the backend will then read a truncated signature and, correctly, drop the connection.Acceptance
enable_proxy_protocol 2on a plaintext listener behind us, connects a client, and asserts that the broker's log records the client's address rather than ours. Mosquitto is worth the test container rather than a hand-rolled parser, since the point of the exercise is that a real broker accepts what we emit.example/default.ymlandREADME.mdupdated, and the Mosquitto stanza that goes with it written out in full, because the thing being sold here is that it works with Mosquitto rather than that the bytes are correct.Notes
hitch has had this since well before 1.7.0, with
write-proxy-v1,write-proxy-v2,write-ipandproxy-proxyfor forwarding a header that arrived from further upstream.proxy-proxyis out of scope here; the outbound direction is what unblocks a deployment.