Skip to content

pjmedia: make VoiceProcessingIO other-audio ducking configurable on Apple platforms - #5178

Open
laconicman wants to merge 1 commit into
pjsip:masterfrom
laconicman:feat/coreaudio-vpio-ducking
Open

pjmedia: make VoiceProcessingIO other-audio ducking configurable on Apple platforms#5178
laconicman wants to merge 1 commit into
pjsip:masterfrom
laconicman:feat/coreaudio-vpio-ducking

Conversation

@laconicman

Copy link
Copy Markdown
Contributor

Fixes #5177

Motivation

The coreaudio backend opens the VoiceProcessingIO (VPIO) audio unit whenever echo cancellation
is enabled. VPIO ducks "other audio" — every audio stream on the system that is not the voice
chat — to improve the intelligibility of the call.

With the system defaults that duck is fixed and held for the entire call. A user listening
to music who accepts a call has that music attenuated from answer to hangup, even during the
long stretches when nobody is talking.

macOS 14 / iOS 17 added kAUVoiceIOProperty_OtherAudioDuckingConfiguration, which exposes two
independent controls over that behaviour:

  • mEnableAdvancedDucking — the style of ducking. When enabled, the duck follows the voice
    activity of the local and remote chat participants: more ducking while someone is talking,
    less when nobody is. Apple compares it to FaceTime SharePlay, where media volume comes back
    up between utterances.
  • mDuckingLevel — the amount of ducking: Default, Min, Mid, Max.

pjproject currently sets neither, so it gets Apple's implicit default: advanced ducking off,
level Default.

Change

create_audio_unit() sets the property right after the audio unit is instantiated, only when
EC is enabled (i.e. only when the unit actually is VPIO). Gating follows the existing in-tree
style for macOS 14 / iOS 17 API in pjmedia/src/pjmedia-videodev/darwin_dev.m: an SDK check
(__IPHONE_17_0 / __MAC_14_0) plus a runtime if (@available(macOS 14.0, iOS 17.0, *)).

The property is queried with AudioUnitGetPropertyInfo first and skipped unless it is present
and writable. Every failure path logs a PJ_LOG(4, …) warning and returns — configuring the
duck must never be able to break audio.

Two config.h macros, #ifndef-guarded so they can be overridden from config_site.h:

Macro Default Effect
PJMEDIA_AUDIO_DEV_COREAUDIO_ADVANCED_DUCKING 1 Enable voice-activity-driven ducking
PJMEDIA_AUDIO_DEV_COREAUDIO_DUCKING_LEVEL 0 (kAUVoiceIOOtherAudioDuckingLevelDefault) Ducking depth

No new pjmedia_aud_dev_cap: this has no independent runtime axis — it is a refinement of
existing EC/VPIO behaviour, not something an application would toggle per call through
pjmedia_aud_stream_set_cap() — and adding one would push PJMEDIA_AUD_DEV_CAP_MAX past its
current 16384.

Why these defaults

Advanced ducking on. The complaint this addresses is precisely "other audio is attenuated
for the whole call". Following voice activity fixes exactly that, and it is the part of the
change that is unambiguously better for any VoIP application.

Depth left at Apple's default. Ducking depth is a much more opinionated axis, and real
deployments differ: a dispatch or contact-centre application may want a deep duck for
intelligibility, a consumer softphone a light one. Apple describes Default as the tuned value
for a typical voice chat, and it is the level pjproject already gets today. A library should
not pick a side there, so it is exposed as its own macro and left where it is. Keeping the
depth unchanged also holds the behavioural delta to one clearly-motivated axis.

Three things worth flagging

1. This changes existing behaviour. Anyone building against a macOS 14+ / iOS 17+ SDK with
EC enabled gets voice-activity-driven ducking instead of a constant duck. Setting
PJMEDIA_AUDIO_DEV_COREAUDIO_ADVANCED_DUCKING to 0 restores the previous behaviour exactly:
with both macros at 0 the property is set to {advanced off, level Default}, which
AudioUnitProperties.h documents as identical to never setting the property at all — "If not
set, the default ducking configuration is to disable advanced ducking, with a ducking level set
to kAUVoiceIOOtherAudioDuckingLevelDefault." Happy to flip the default to opt-in (0) if you
would rather ship this as strictly additive.

2. mDuckingLevel is meaningful whether or not advanced ducking is on. I checked rather
than assumed. WWDC23 session 10235 states the struct "provides controls of two independent
aspects of ducking — the style of ducking, that is mEnableAdvancedDucking, and the amount of
ducking, that is mDuckingLevel", and that "the two controls can be used independently".
That is why they are two macros and not one enum.

3. Platform scope. kAUVoiceIOProperty_OtherAudioDuckingConfiguration is declared
API_AVAILABLE(ios(17.0), macos(14.0)) API_UNAVAILABLE(watchos, tvos). That covers every
platform this backend targets — macOS, iOS, and Mac Catalyst, which inherits the iOS
availability — and I verified Catalyst compiles. It does not cover tvOS or watchOS, where
the symbol is declared but unavailable and would fail to compile. The guard I used matches the
file's convention that TARGET_OS_IPHONE means iOS, which is true for pjproject today (there
is no tvOS or watchOS build support anywhere in the tree). If you would rather harden it
against a future port, && !TARGET_OS_TV && !TARGET_OS_WATCH on the TARGET_OS_IPHONE arm is
the one-line change — say the word and I will add it.

Validation

  • make in pjmedia/build on aarch64-apple-darwin — clean, zero warnings, and the new code
    is confirmed present in the built coreaudio_dev.o.
  • coreaudio_dev.m compiles clean (-Wall -Wextra, no output) for: macOS at the host
    deployment target and at -mmacosx-version-min=10.15; iOS arm64 at deployment targets 15.0
    and 12.0 — i.e. both the compiled-in and the runtime-@available-false paths; Mac Catalyst
    arm64; with PJMEDIA_AUDIO_DEV_COREAUDIO_ADVANCED_DUCKING=0; with
    PJMEDIA_AUDIO_DEV_COREAUDIO_DUCKING_LEVEL=kAUVoiceIOOtherAudioDuckingLevelMin (confirming
    the macro accepts the Apple constants by name from config_site.h); and with
    PJMEDIA_AUDIO_DEV_HAS_COREAUDIO=0.
  • Non-Apple platforms are unaffected: the two new macros are referenced only in
    coreaudio_dev.m, and config.h gains only integer literals, so it still compiles with no
    Apple headers in scope (verified by compiling audiodev.c against it).
  • Not verified: the listening test. I have not played music, placed a call with EC enabled,
    and confirmed by ear that the music is no longer flattened for the call's duration and that
    echo cancellation still works. That check is human-only and has not been done.

…pple platforms

The coreaudio backend opens the VoiceProcessingIO audio unit whenever echo
cancellation is enabled. With the system defaults the OS applies a fixed duck
to every other audio source for the whole duration of the call, so a user
listening to music who takes a call has that music attenuated until hangup.

macOS 14 / iOS 17 added kAUVoiceIOProperty_OtherAudioDuckingConfiguration,
which exposes two independent axes: advanced ducking, which follows the voice
activity of the chat participants instead of holding a constant duck, and the
ducking level.

Set the property after the audio unit is instantiated, only when EC is enabled,
guarded by both an SDK check and a runtime @available check. The property is
queried with AudioUnitGetPropertyInfo first, and any failure only logs a
warning so audio is never broken by it.

Two config.h macros control it:

- PJMEDIA_AUDIO_DEV_COREAUDIO_ADVANCED_DUCKING (default 1) enables advanced
  ducking. Set it to 0 for the previous behaviour.
- PJMEDIA_AUDIO_DEV_COREAUDIO_DUCKING_LEVEL (default
  kAUVoiceIOOtherAudioDuckingLevelDefault) sets the ducking depth, which is
  left at Apple's tuned value for a typical voice chat.
@sauwming

Copy link
Copy Markdown
Member

1. This changes existing behaviour. Happy to flip the default to opt-in (0) if you would rather ship this as strictly additive.

Yes, I would prefer to leave the existing behaviour unchanged by setting the default as 0.
What I can offer you is to enable PJMEDIA_AUDIO_DEV_COREAUDIO_ADVANCED_DUCKING to 1 inside config_site_sample.h instead (within PJ_CONFIG_IPHONE).

3. Platform scope. If you would rather harden it against a future port,&& !TARGET_OS_TV && !TARGET_OS_WATCHon theTARGET_OS_IPHONE` arm is the one-line change — say the word and I will add it.

This also sounds good. Yes, please add it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

coreaudio: VPIO ducks other audio for the whole call, with no way to configure it

2 participants