test: exercise Flutter compliance through the native macOS bridge

[dustinbyrne]

💡 Motivation and Context

Replace the compliance adapter's handwritten Dart transport with a real macOS Flutter application using the public facade, shipped MethodChannel and registered Apple plugin. The native SDK now owns payloads, UUIDs, retries, flag results and side effects.

Each harness initialization launches the app in a fresh temporary home so pending native events cannot leak between cases. A passive proxy observes bytes without retrying. Missing timestamp, state and unprovable flush controls remain explicit failures. CI runs the unchanged harness 1.0.0 source natively and requires the complete 47-case inventory, while keeping compliance assertions advisory.

Scope is macOS-first: this does not certify Android, iOS devices or web. SDK APIs, shipping source, defaults and dependency contracts are unchanged; no package release is needed.

💚 How did you test it?

• Built the real Flutter 3.44.0 / Dart 3.12 release app against wrapper 5.39.0 and resolved Apple SDK 3.71.6.
• Three observer tests and Dart analysis passed.
• Real-app regression passed: fresh-home reset isolates queued events, capture retries preserve UUID/timestamp, and public flags reload/getter exercises 502/504 retries and SDK-generated called events.
• Full fresh-process harness: 47 selected, 14 passed, 33 failed (capture 12/30; flags 2/17). Failures remain visible: identity/group side effects affect generic request/count fixtures, Apple requires flag payload metadata omitted by some fixtures, mobile payload fields differ, and unsupported timestamp/drain controls fail explicitly. Some passing generic assertions also see incidental SDK requests; this is not a general native conformance claim.
• Verified all 47 IDs, rejection of empty reports, and byte-identical shipping SDK files in the built workspace. CI results and resolved delegates are uploaded separately.

📝 Checklist

• I reviewed the submitted code.
• I added tests to verify the changes.
• I updated the docs if needed.
• No breaking change or entry added to the changelog.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Implemented with a Pi worker using file tools, Bash, Git and GitHub CLI; a fresh read-only reviewer inspected the same candidate. Native process isolation keeps the real Flutter entry point and SDK persistence behavior intact. Human review is still required.

[dustinbyrne]

SDK compliance report — all CI profiles

SDK head: 22831ad5477ecba2f2425c8367cb5ead57199627
CI checkout: 05b8f103a3f85b26b49250d3bb724bdee505b3b7
Harness: 1.0.0 / contract 1.2.

These reports are from the completed CI artifacts linked below. All selected results are included; assertion failures remain advisory and are not hidden by successful workflow checks.

Profile	Passed / selected	Failed	Evidence
macOS	14/47	33	CI run

Real macOS Flutter app → public Dart facade → shipped MethodChannel/Apple plugin, using a fresh owned process/home per init. This is not iOS-device, Android or web coverage. Native identity/reload traffic and shared fixtures affect several assertions.

macOS: 14/47 passed, 33 failed — full report

posthog-flutter Compliance Report

Date: 2026-09-08T15:00:32.278409+00:00
Duration: 171561ms

⚠️ Some Tests Failed

14/47 tests passed, 33 failed

---

Capture Tests

⚠️ 12/30 tests passed, 18 failed

View Details

Test	Status	Duration
Format Validation.Event Has Required Fields	❌	2547ms
Format Validation.Event Has Uuid	❌	1752ms
Format Validation.Event Has Lib Properties	✅	1620ms
Format Validation.Distinct Id Is String	✅	1355ms
Format Validation.Token Is Present	✅	1446ms
Format Validation.Custom Properties Preserved	❌	1392ms
Format Validation.Event Has Timestamp	❌	1602ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc	❌	539ms
Retry Behavior.Retries On 503	✅	7546ms
Retry Behavior.Does Not Retry On 400	❌	3519ms
Retry Behavior.Does Not Retry On 401	❌	3375ms
Retry Behavior.Respects Retry After Header	❌	6577ms
Retry Behavior.Implements Backoff	❌	20291ms
Retry Behavior.Retries On 500	✅	7769ms
Retry Behavior.Retries On 502	✅	6911ms
Retry Behavior.Retries On 504	✅	6817ms
Retry Behavior.Max Retries Respected	❌	30477ms
Deduplication.Generates Unique Uuids	✅	2894ms
Deduplication.Preserves Uuid On Retry	❌	6983ms
Deduplication.Preserves Uuid And Timestamp On Retry	❌	13895ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry	❌	8505ms
Deduplication.No Duplicate Events In Batch	✅	2168ms
Deduplication.Different Events Have Different Uuids	✅	1636ms
Compression.Sends Gzip When Enabled	✅	1774ms
Batch Format.Uses Proper Batch Structure	❌	1797ms
Batch Format.Flush With No Events Sends Nothing	❌	579ms
Batch Format.Multiple Events Batched Together	❌	1625ms
Error Handling.Does Not Retry On 403	❌	3651ms
Error Handling.Does Not Retry On 413	❌	3835ms
Error Handling.Retries On 408	✅	6754ms

Failures

format_validation.event_has_required_fields

Expected 1 requests, got 3

format_validation.event_has_uuid

Event missing 'uuid' field

format_validation.custom_properties_preserved

Expected custom_string='hello', got 'None'. Available properties: ['$locale', '$os_version', '$process_person_profile', '$session_id', '$is_testflight', '$os_name', '$lib', '$timezone', 'distinct_id', '$is_sideloaded']

format_validation.event_has_timestamp

Event missing 'timestamp' field

format_validation.non_utc_event_timestamp_is_converted_to_utc

501, message='Not Implemented', url='http://127.0.0.1:18310/capture'

retry_behavior.does_not_retry_on_400

Expected 1 requests, got 2

retry_behavior.does_not_retry_on_401

Expected 1 requests, got 2

retry_behavior.respects_retry_after_header

Retry delay too short: 2ms < 2500ms

retry_behavior.implements_backoff

First retry delay too short: 1ms < 100ms

retry_behavior.max_retries_respected

500, message='Internal Server Error', url='http://127.0.0.1:18310/flush'

deduplication.preserves_uuid_on_retry

UUIDs changed on retry: [] != ['01a08188-0528-7746-a809-16eff9280474', '01a08188-052e-7172-8619-15ec8cc73eb6']

deduplication.preserves_uuid_and_timestamp_on_retry

UUIDs changed on retry: ['01a08188-204d-75f0-982e-09d5047f4a6a', '01a08188-205d-77c2-9ab3-2856f0196775'] != []

deduplication.preserves_uuid_and_timestamp_on_batch_retry

UUIDs changed on retry: [] != ['01a08188-5865-7aa2-8c21-cd21b1af4813', '01a08188-5877-7131-9e68-1954085a606e']

batch_format.uses_proper_batch_structure

Batch format missing 'batch' array field

batch_format.flush_with_no_events_sends_nothing

501, message='Not Implemented', url='http://127.0.0.1:18310/flush'

batch_format.multiple_events_batched_together

Expected 1 requests, got 2

error_handling.does_not_retry_on_403

Expected 1 requests, got 2

error_handling.does_not_retry_on_413

Expected 1 requests, got 2

Feature_Flags Tests

⚠️ 2/17 tests passed, 15 failed

View Details

Test	Status	Duration
Request Payload.Request With Person Properties Device Id	❌	678ms
Request Payload.Flags Request Uses V2 Query Param	❌	403ms
Request Payload.Flags Request Hits Flags Path Not Decide	❌	322ms
Request Payload.Flags Request Omits Authorization Header	❌	355ms
Request Payload.Token In Flags Body Matches Init	✅	476ms
Request Payload.Groups Round Trip	❌	313ms
Request Payload.Groups Default To Empty Object	❌	318ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False	❌	497ms
Request Payload.Disable Geoip Omitted Defaults To False	❌	360ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key	❌	417ms
Request Lifecycle.No Flags Request On Init Alone	✅	261ms
Request Lifecycle.No Flags Request On Normal Capture	❌	1393ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests	❌	925ms
Request Lifecycle.Mock Response Value Is Returned To Caller	❌	495ms
Retry Behavior.Retries Flags On 502	❌	894ms
Retry Behavior.Retries Flags On 504	❌	1046ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event	❌	573ms

Failures

request_payload.request_with_person_properties_device_id

Expected 1 /flags requests, got 2

request_payload.flags_request_uses_v2_query_param

Expected 1 /flags requests, got 2

request_payload.flags_request_hits_flags_path_not_decide

Expected 1 /flags requests, got 2

request_payload.flags_request_omits_authorization_header

Expected 1 /flags requests, got 2

request_payload.groups_round_trip

Field 'company' not found in /flags request body at path 'groups.company'. Available keys: []

request_payload.groups_default_to_empty_object

Field 'group_properties' not found in /flags request body at path 'group_properties'. Available keys: ['person_properties', '$device_id', 'timezone', 'distinct_id', 'groups', 'api_key', '$anon_distinct_id']

request_payload.disable_geoip_false_propagates_as_geoip_disable_false

Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['timezone', '$device_id', 'distinct_id', 'groups', 'person_properties', 'api_key', '$anon_distinct_id']

request_payload.disable_geoip_omitted_defaults_to_false

Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['$device_id', 'api_key', 'distinct_id', 'timezone', '$anon_distinct_id', 'person_properties', 'groups']

request_payload.flag_keys_to_evaluate_contains_only_requested_key

Field 'flag_keys_to_evaluate' not found in /flags request body at path 'flag_keys_to_evaluate'. Available keys: ['distinct_id', 'groups', '$device_id', '$anon_distinct_id', 'person_properties', 'timezone', 'api_key']

request_lifecycle.no_flags_request_on_normal_capture

Expected 0 /flags requests, got 1

request_lifecycle.two_flag_calls_produce_two_remote_requests

Expected 2 /flags requests, got 3

request_lifecycle.mock_response_value_is_returned_to_caller

Expected last action result 'variant-a', got None

retry_behavior.retries_flags_on_502

Expected 2 /flags requests, got 3

retry_behavior.retries_flags_on_504

Expected 2 /flags requests, got 3

side_effect_events.get_feature_flag_captures_feature_flag_called_event

501, message='Not Implemented', url='http://127.0.0.1:18310/flush'