Skip to content

Mint adapter: FunctionClauseError in ConnectionProcess.process_response/2 on GOAWAY racing in-flight requests — kills the channel permanently #563

Description

@freevova

Describe the bug
With the Mint adapter, a server GOAWAY that races in-flight requests crashes the whole ConnectionProcess with a FunctionClauseError instead of failing just the raced requests. The
crash is an abnormal exit, so it bypasses both the adapter's retry/attempt_reconnect loop and the {:elixir_grpc, :connection_down, pid} parent notification (both only run on the
graceful-disconnect path). The channel owner only gets a linked {:EXIT, pid, {:function_clause, ...}} message (connect/2 traps exits on the caller), so unless it handles that
explicitly it keeps serving a channel whose conn_pid is dead — every subsequent call fails with :noproc, and callers that had requests in flight exit or hang.

Mechanism: Mint.HTTP2.stream/2 returns requests that were in flight when GOAWAY arrived as {:error, request_ref, %Mint.HTTPError{reason: :unprocessed}} entries in the responses list
(documented Mint behavior). ConnectionProcess.process_response/2 only has clauses for :status, :headers, :data and :done, so the error entry raises.

To Reproduce
Steps to reproduce the behavior:

  1. Connect with the Mint adapter to an HTTP/2 server that recycles idle connections with GOAWAY (e.g. an nginx ingress with default keepalive settings, ~60s idle).
  2. Keep a steady stream of unary calls so requests are in flight when GOAWAY lands.
  3. The race eventually produces the crash below. Deterministic single-stream reproduction: inject a raw RST_STREAM frame for an in-flight request — same response-entry shape (see the
    regression test in fix(mint): handle stream-level errors in process_response instead of crashing #562).

Expected behavior
The raced requests fail with a per-request error delivered to their stream response processes (:unprocessed is retryable by contract — the server did not process the request), and the
connection process stays alive so connection-level GOAWAY handling goes through the existing graceful paths.

Logs

** (FunctionClauseError) no function clause matching in GRPC.Client.Adapters.Mint.ConnectionProcess.process_response/2
    (grpc 1.0.2) lib/grpc/client/adapters/mint/connection_process/connection_process.ex:248:
        GRPC.Client.Adapters.Mint.ConnectionProcess.process_response(
          {:error, #Reference<...>, %Mint.HTTPError{reason: :unprocessed, module: Mint.HTTP2}},
          %GRPC.Client.Adapters.Mint.ConnectionProcess.State{conn: %Mint.HTTP2{state: {:goaway, :no_error, ""}, ...}, ...})
    (grpc 1.0.2) lib/grpc/client/adapters/mint/connection_process/connection_process.ex:212:
        GRPC.Client.Adapters.Mint.ConnectionProcess.handle_info/2

Protos
Not proto-specific — any unary call in flight at GOAWAY time triggers it.

Versions:

  • OS: Linux (Docker on AWS EKS)
  • Erlang: OTP 28
  • Elixir: 1.20.2
  • mix.lock(grpc, gun, cowboy, cowlib): grpc 1.0.2 (hex), grpc_core 1.0.2, gun not installed (Mint adapter), cowboy 2.17.0, cowlib 2.18.0

Additional context
In production behind an nginx ingress this fires several times per hour under steady traffic; each occurrence kills or wedges the background jobs whose calls were in flight. Fix proposed
in #562 (adds the missing process_response({:error, request_ref, error}, state) clause, mirroring the connection-close path; includes a deterministic regression test).

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions