Skip to content

feat(sdk): add active-tool audio support#119

Open
brunoro wants to merge 5 commits into
mainfrom
feat/audio
Open

feat(sdk): add active-tool audio support#119
brunoro wants to merge 5 commits into
mainfrom
feat/audio

Conversation

@brunoro

@brunoro brunoro commented Jul 22, 2026

Copy link
Copy Markdown

Adds foreground audio support to the SDK client through a small, opinionated API built on Android's audio libraries:

  • LightAudio creates the different audio components.
  • LightAudioPlayer provides queue-based playback for files, assets, and URLs using androidx.media3.
  • LightAudioRecorder records microphone input to MPEG-4 files with AAC audio using android.media.MediaRecorder
  • LightAudioCapture exposes real-time microphone input as a Flow of PCM buffers using android.media.AudioRecord
  • LightAudioVoice plays short PCM buffers for synthesized sounds and samples using android.media.AudioTrack
  • Focus handling through android.media.AudioManager.
  • Add android.permission.RECORD_AUDIO to tool permission allowlist.

Persistent audio (that continues playing after the tool is left) support is not included in this first public surface. Since it requires a bit of additional work on LightOS, we'll first focus on getting the active-tool API stable.

The audio-demo example showcases the usage of the library:
audio-demo
audio-demo-player
audio-demo-recorder
audio-demo-capture
audio-demo-voice

Other minor changes:

  • Adds optional orientation field to TOML config to define tool screen orientation
  • Sets minSdk level to 34.

@brunoro brunoro changed the title feat(sdk): add foreground audio support feat(sdk): add active-tool audio support Jul 22, 2026
@jonathancaudill

Copy link
Copy Markdown

Okay, I think this is an awesome foundation.

Few things I'd like to note from a practicality and end-user ux standpoint (ignoring the foreground service stuff, which you folks are already working on lol):

This PR would introduce an internal ExoPlayer instance managed entirely by the SDK. Which is great for 99.9% of use cases, but music streaming is a tricky business 😭

The nice thing is that media3 already has a pretty decent streaming pipeline. That's what the TIDAL half of phono uses (since TIDAL serves raw streaming URLs). I can definitely rework the Spotify half towards that. Currently we have a fully custom sink, since Spotify serves encrypted chunks. But it would absolutely be possible to decrypt, cache, etc., then feed directly to media3 as a custom source.

The problem at the moment is that media3's proper streaming capabilities live in ExoPlayer.Builder, which is walled off. While it would be possible to just feed through some of these params as Light-denoted APIs, I think that's probably not the best long-term solution since devs may need more over time, and since it increases surface area for breaking changes from exoplayer itself

I would ask that the SDK allow handoff of pre-configured ExoPlayer instances, instead of LightOS always making its own. Even if LightOS were to copy the config into a new ExoPlayer build, the cache and priority setup we would be using to properly stream decrypted ogg wouldn't come along.

AudioAttributes, AudioSink, focus listening, pause/play/etc., would remain gated, but the tool can configure the input half (source factory, cache, priority, load control). SDK can own and reassert sink, attributes, focus, lifecycle after it adopts the player.

I'm happy to prototype the handoff here. The TIDAL backend runs a fully configured media3 player already, so would prob be a low lift to work and post some proto-code here

@jonathancaudill

jonathancaudill commented Jul 23, 2026

Copy link
Copy Markdown

As an aside, it's really exciting to collaborate on the sdk with you guys. Community has been proposing more backends for phono, which could mean a mostly unified streaming client, which is pretty sick.

lmk if you'd like me to make a PR w/ a rough prototype of how this could all work

EDIT: realizing that the above two posts sound a little AI-ish. I wrote them myself I promise lol

@dupontgu

dupontgu commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

@jonathancaudill At the highest level, you're saying you could get what you need by allowing devs to inject an exoplayer instance/factory instead of using our default? I'd definitely be open to that as long as it doesn't make things any more complex for the 99.9% of cases. I'm gonna stack a PR on top of this with some kotlin'y stuff in just a bit here but after that, feel free to open your own. (we're not gonna wait to merge this but it should still be safe to work off of after my changes go in)

Edit: my changes are in #123 - want to test them in the morning before I merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants