Enrich `Wait For Request` / `Wait For Response`: metadata, binary body support, and sibling consistency

[Snooz82]

Closes #3320 (binary response body, priority: high).

Use case

Wait For Request and Wait For Response return dicts that users assert on, but three distinct problems limit them today:

1. Missing metadata. Several pieces of data Playwright already holds on the captured Request/Response objects are not copied into the dicts:

• set-cookie cannot be asserted at all. headers comes from Playwright's headers(), which deliberately omits security-related headers such as set-cookie. Verifying that a login response actually set a session cookie has no workaround via this keyword. allHeaders() fixes this.
• Requests: resource type (xhr, fetch, image, …), navigation flag, redirect origin, timing/sizes — e.g. "the API call finished under 500 ms".
• Responses: HTTP version, server address, TLS details, served from a service worker.

2. Binary bodies are destroyed (#3320). The wrapper reads every response body with data.text() (node/playwright-wrapper/network.ts:79); Playwright's response.body() (Buffer) is never used. A user intercepting an image gets UTF-8-mangled text that cannot be converted back to bytes. The reporter's current workaround is re-issuing the captured request with Python requests — which defeats the purpose of interception (second request, different session). aaltat's guidance in the issue: let the user choose text vs. bytes.

3. The two sibling keywords are inconsistent. Users reasonably expect "a request" to look the same everywhere, but today:

	Wait For Request (flat dict)	request nested in Wait For Response
url	present	missing (only the response's url exists)
postData parsing	JSON.parse attempted unconditionally (node side, network.ts:122)	parsed only if request content-type is application/json (Python side, network.py:43 _jsonize_content)
headers transport	plain dict	JSON-string, re-parsed Python-side

Additionally, the response body is auto-JSON-parsed regardless of content-type (network.py:141 tries json.loads on every body), so users can neither rely on a documented rule nor opt out of the parsing magic.

We cannot silently "fix" the parsing rules — thousands of suites depend on the current shapes. The design below adds consistency additively and gives users an explicit opt-in switch, so nothing existing changes behavior.

Proposed changes

a) Additive metadata keys (no signature change)

Wait For Request dict gains: resourceType, isNavigationRequest, redirectedFrom (URL or None), timing (dict), sizes (dict), allHeaders (incl. browser-added headers).

Wait For Response dict gains: allHeaders (including set-cookie), httpVersion, serverAddr ({ipAddress, port} or None), securityDetails (dict or None), fromServiceWorker (bool), timing.

Sibling consistency: the nested request dict inside Wait For Response gains the same request-side keys with the same names and semantics as the flat Wait For Request dict — including the currently missing url. From then on, every documented request-side key exists in both places; only the legacy postData parsing difference remains (kept as-is, but now explicitly documented in both keyword docs, see c).

b) body_format= argument on Wait For Response (solves #3320)

New named-only argument:

Wait For Response matcher timeout=None *, body_format=AUTO

• AUTO (default — exactly today's behavior): body transferred as text, JSON-parsed to a dict when parseable.
• TEXT: body as plain str, no JSON auto-parsing (documented opt-out of the magic).
• BYTES: body returned as Python bytes, read via response.body() and transferred as raw bytes — binary-safe. No JSON parsing.
• NONE: body is None, transfer skipped entirely (multi-MB payloads no longer cross the gRPC boundary just to be thrown away).

*** Test Cases ***
Login Sets Session Cookie
    ${promise}=    Promise To    Wait For Response    matcher=**/api/login
    Click    id=login-button
    ${response}=    Wait For    ${promise}
    Should Contain    ${response.allHeaders}[set-cookie]    sessionid=
    Should Be Equal    ${response.httpVersion}    h2

Save Intercepted Image    # the #3320 scenario
    ${promise}=    Promise To    Wait For Response    matcher=**/logo.png    body_format=BYTES
    Go To    ${URL}
    ${response}=    Wait For    ${promise}
    ${path}=    Evaluate    pathlib.Path('${OUTPUT_DIR}/logo.png').write_bytes($response.body)    modules=pathlib

API Call Is Fast Fetch
    ${promise}=    Promise To    Wait For Request    matcher=**/api/data
    Click    id=load-data
    ${request}=    Wait For    ${promise}
    Should Be Equal    ${request.resourceType}    fetch
    Should Be True    ${request.timing.responseEnd} < 500

The same enum can later be reused for a post_data_format= argument on Wait For Request (Playwright offers request.postDataBuffer()); out of scope here but the naming should anticipate it.

c) Documentation of the legacy quirks

The keyword docs of both siblings get an explicit "Parsing rules" section stating: how postData is parsed in each place, that body_format=AUTO JSON-parses regardless of content-type, and that headers omits security headers while allHeaders does not. This costs nothing and removes the surprise factor for the differences we deliberately keep.

Migration / mitigation story

• No behavior change without opt-in. All defaults (body_format=AUTO, existing keys, existing parsing) stay byte-for-byte identical. New keys are additive; DotDict access to existing keys is unaffected.
• Users hit by the quirks get an explicit escape hatch instead of a silent change: body_format=TEXT|BYTES replaces guessing, allHeaders replaces missing headers, nested request.url removes the need to correlate flat and nested shapes.
• If a future major release ever wants to unify the postData parsing rules, the path is already laid: introduce post_data_format= with the same enum, default AUTO documenting current behavior — never a silent flip of defaults.

Playwright API

Request side: request.resourceType, request.isNavigationRequest, request.redirectedFrom, request.timing, request.sizes, request.allHeaders.

Response side: response.allHeaders, response.body, response.httpVersion, response.serverAddr, response.securityDetails, response.fromServiceWorker.

Implementation notes

• protobuf/playwright.proto: Request.HttpCapture gains a bodyFormat field; the streamed Response.Json chunk message gains a bytes bodyPartBytes field alongside the existing string bodyPart (additive proto changes). Raw protobuf bytes chunks avoid the ~33 % base64 overhead; base64 through the existing string field is the fallback option if proto changes are to be avoided.
• node/playwright-wrapper/network.ts: waitForResponse branches on bodyFormat — response.body() Buffer chunked into bodyPartBytes for BYTES, current data.text() + splitUtf8ByMaxBytes path for AUTO/TEXT, no body read for NONE. Metadata fields added to the serialized JSON (async allHeaders/sizes/serverAddr/securityDetails awaited here). waitForRequest gains the same metadata fields; the nested request dict in waitForResponse is built from the same serializer for consistency.
• Browser/keywords/network.py: _wait_for_http_response assembles bytes from the byte chunks and skips both JSON-parse steps for BYTES/TEXT/NONE; new BodyFormat enum in Browser/utils/data_types.py; docs for both keywords incl. the "Parsing rules" section.
• atest: set-cookie assertion, redirect metadata, a binary endpoint in the dynamic test app (PNG) round-tripped via body_format=BYTES and compared byte-for-byte, NONE returning body=None, and a regression test that AUTO output is unchanged.

Backwards compatibility

Fully backwards compatible by construction: no signature changes except one new named-only argument whose default reproduces today's behavior exactly; returned DotDicts only gain keys; wire changes are additive proto fields. The known shape inconsistencies between the siblings are documented rather than silently changed; any future unification goes through explicit opt-in arguments, never through changed defaults.